Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-31 Thread Roland Fehrenbacher 
> "Nathan" == Nathan Hjelm  writes:

Hi Nathan

Nathan> I want to close the loop on this issue. 1.8.5 will address
Nathan> it in several ways:

Nathan>  - knem support in btl/sm has been fixed. A sanity check was
Nathan>disabling knem during component registration. I wrote the
Nathan>sanity check before the 1.7 release and didn't intend
Nathan>this side-effect.

Nathan>  - vader now supports xpmem, cma, and knem. The best
Nathan>available single-copy mechanism will be used. If multiple
Nathan>single-copy mechanisms are available you can select which
Nathan>one you want to use are runtime.

Nathan> More about the vader btl can be found here:
Nathan> 
http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy/

Nathan> -Nathan Hjelm HPC-5, LANL

thanks for the great info. Question about xpmem: Are there any plans by
someone to maintain the code?

Roland

---
http://www.q-leap.com / http://qlustar.com
  --- HPC / Storage / Cloud Linux Cluster OS ---

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-31 Thread Brice Goglin 
Le 31/10/2014 00:24, Gus Correa a écrit :
> 2) Any recommendation for the values of the
> various vader btl parameters?
> [There are 12 of them in OMPI 1.8.3!
> That is real challenge to get right.]
>
> Which values did you use in your benchmarks?
> Defaults?
> Other?
>
> In particular, is there an optimal value for the eager/rendevous
> threshold value? (btl_vader_eager_limit, default=4kB)
> [The INRIA web site suggests 32kB for the sm+knem counterpart
> (btl_sm_eager_limit, default=4kB).]

There's no perfect value, and no easy way to tune all this.

The impact of direct copy mechanisms such as XPMEM/KNEM/CMA depends on
the contention in your memory bus and caches. If you're doing a
Alltoall, the optimal threshold for enabling them will be much lower
than if you're doing a pingpong because doing a single copy instead of
two usually helps more when the memory subsystem is overloaded. And it
also depends on your process placement and what cache (and cache size)
is shared between them.

Unfortunately, microbenchmarks will hardly help you decide of a better
threshold because performance also depend on the state of buffers in
caches (did the application writes the send buffer recently? will the
application read the buffer soon? microbenchmark ignore these), and each
copy strategy may have different impact of caches (which process is
reading and writing from which processors and from/to which buffer?).

So I'd say don't bother tuning things for too long...

Brice

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-30 Thread Gus Correa 

On 10/30/2014 07:32 PM, Ralph Castain wrote:

Just for FYI: I believe Nathan misspoke.
The new capability is in 1.8.4, which I hope
to release next Friday (Nov 7th)



Hi Ralph

That is even better!
Look forward to OMPI 1.8.4.

I still would love to hear from Nathan / OMPI team
about my remaining questions below
(specially the 12 vader parameters).

Many thanks,
Gus Correa


On Oct 30, 2014, at 4:24 PM, Gus Correa  wrote:

Hi Nathan

Thank you very much for addressing this problem.

I read your notes on Jeff's blog about vader,
and that clarified many things that were obscure to me
when I first started this thread
whining that knem was not working in OMPI 1.8.3.
Thank you also for writing that blog post,
and for sending the link to it.
That was very helpful indeed.

As your closing comments on the blog post point out,
and your IMB benchmark graphs of pingpong/latency &
sendrecv/bandwidth show,
vader+xpmem outperforms the other combinations
of btl+memory_copy_mechanism of intra-node communication.

For the benefit of pedestrian OpenMPI users like me:

1) What is the status of xpmem in the Linux world at this point?
[Proprietary (SGI?) / open source, part of the Linux kernel (which),
part of standard distributions (which) ?]

2) Any recommendation for the values of the
various vader btl parameters?
[There are 12 of them in OMPI 1.8.3!
That is real challenge to get right.]

Which values did you use in your benchmarks?
Defaults?
Other?

In particular, is there an optimal value for the eager/rendevous threshold 
value? (btl_vader_eager_limit, default=4kB)
[The INRIA web site suggests 32kB for the sm+knem counterpart 
(btl_sm_eager_limit, default=4kB).]

3) Did I understand it right, that the upcoming OpenMPI 1.8.5
can be configured with more than one memory copy mechanism altogether
(e.g. --with-knem and --with-cma and --with-xpmem),
then select one of them at runtime with the btl_vader_single_copy_mechanism 
parameter?
Or must OMPI be configured with only one memory copy mechanism?

Many thanks,
Gus Correa


On 10/30/2014 05:44 PM, Nathan Hjelm wrote:

I want to close the loop on this issue. 1.8.5 will address it in several
ways:

  - knem support in btl/sm has been fixed. A sanity check was disabling
knem during component registration. I wrote the sanity check before
the 1.7 release and didn't intend this side-effect.

  - vader now supports xpmem, cma, and knem. The best available
single-copy mechanism will be used. If multiple single-copy
mechanisms are available you can select which one you want to use are
runtime.

More about the vader btl can be found here:
http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy/

-Nathan Hjelm
HPC-5, LANL

On Fri, Oct 17, 2014 at 01:02:23PM -0700, Ralph Castain wrote:

  On Oct 17, 2014, at 12:06 PM, Gus Correa  wrote:
  Hi Jeff

  Many thanks for looking into this and filing a bug report at 11:16PM!

  Thanks to Aurelien, Ralph and Nathan for their help and clarifications
  also.

  **

  Related suggestion:

  Add a note to the FAQ explaining that in OMPI 1.8
  the new (default) btl is vader (and what it is).

  It was a real surprise to me.
  If Aurelien Bouteiller didn't tell me about vader,
  I might have never realized it even existed.

  That could be part of one of the already existent FAQs
  explaining how to select the btl.

  **

  Doubts (btl in OMPI 1.8):

  I still don't understand clearly the meaning and scope of vader
  being a "default btl".

We mean that it has a higher priority than the other shared memory
implementation, and so it will be used for intra-node messaging by
default.

  Which is the scope of this default: intra-node btl only perhaps?

Yes - strictly intra-node

  Was there a default btl before vader, and which?

The "sm" btl was the default shared memory transport before vader

  Is vader the intra-node default only (i.e. replaces sm  by default),

Yes

  or does it somehow extend beyond node boundaries, and replaces (or
  brings in) network btls (openib,tcp,etc) ?

Nope - just intra-node

  If I am running on several nodes, and want to use openib, not tcp,
  and, say, use vader, what is the right syntax?

  * nothing (OMPI will figure it out ... but what if you have
  IB,Ethernet,Myrinet,OpenGM, altogether?)

If you have higher-speed connections, we will pick the fastest for
inter-node messaging as the "default" since we expect you would want the
fastest possible transport.

  * -mca btl openib (and vader will come along automatically)

Among the ones you show, this would indeed be the likely choices (openib
and vader)

  * -mca btl openib,self (and vader will come along automatically)

The "self" btl is *always* active as the loopback

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-30 Thread Ralph Castain 
Just for FYI: I believe Nathan misspoke. The new capability is in 1.8.4, which 
I hope to release next Friday (Nov 7th)

> On Oct 30, 2014, at 4:24 PM, Gus Correa  wrote:
> 
> Hi Nathan
> 
> Thank you very much for addressing this problem.
> 
> I read your notes on Jeff's blog about vader,
> and that clarified many things that were obscure to me
> when I first started this thread
> whining that knem was not working in OMPI 1.8.3.
> Thank you also for writing that blog post,
> and for sending the link to it.
> That was very helpful indeed.
> 
> As your closing comments on the blog post point out,
> and your IMB benchmark graphs of pingpong/latency &
> sendrecv/bandwidth show,
> vader+xpmem outperforms the other combinations
> of btl+memory_copy_mechanism of intra-node communication.
> 
> For the benefit of pedestrian OpenMPI users like me:
> 
> 1) What is the status of xpmem in the Linux world at this point?
> [Proprietary (SGI?) / open source, part of the Linux kernel (which),
> part of standard distributions (which) ?]
> 
> 2) Any recommendation for the values of the
> various vader btl parameters?
> [There are 12 of them in OMPI 1.8.3!
> That is real challenge to get right.]
> 
> Which values did you use in your benchmarks?
> Defaults?
> Other?
> 
> In particular, is there an optimal value for the eager/rendevous threshold 
> value? (btl_vader_eager_limit, default=4kB)
> [The INRIA web site suggests 32kB for the sm+knem counterpart 
> (btl_sm_eager_limit, default=4kB).]
> 
> 3) Did I understand it right, that the upcoming OpenMPI 1.8.5
> can be configured with more than one memory copy mechanism altogether
> (e.g. --with-knem and --with-cma and --with-xpmem),
> then select one of them at runtime with the btl_vader_single_copy_mechanism 
> parameter?
> Or must OMPI be configured with only one memory copy mechanism?
> 
> Many thanks,
> Gus Correa
> 
> 
> On 10/30/2014 05:44 PM, Nathan Hjelm wrote:
>> I want to close the loop on this issue. 1.8.5 will address it in several
>> ways:
>> 
>>  - knem support in btl/sm has been fixed. A sanity check was disabling
>>knem during component registration. I wrote the sanity check before
>>the 1.7 release and didn't intend this side-effect.
>> 
>>  - vader now supports xpmem, cma, and knem. The best available
>>single-copy mechanism will be used. If multiple single-copy
>>mechanisms are available you can select which one you want to use are
>>runtime.
>> 
>> More about the vader btl can be found here:
>> http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy/
>> 
>> -Nathan Hjelm
>> HPC-5, LANL
>> 
>> On Fri, Oct 17, 2014 at 01:02:23PM -0700, Ralph Castain wrote:
>>>  On Oct 17, 2014, at 12:06 PM, Gus Correa  
>>> wrote:
>>>  Hi Jeff
>>> 
>>>  Many thanks for looking into this and filing a bug report at 11:16PM!
>>> 
>>>  Thanks to Aurelien, Ralph and Nathan for their help and clarifications
>>>  also.
>>> 
>>>  **
>>> 
>>>  Related suggestion:
>>> 
>>>  Add a note to the FAQ explaining that in OMPI 1.8
>>>  the new (default) btl is vader (and what it is).
>>> 
>>>  It was a real surprise to me.
>>>  If Aurelien Bouteiller didn't tell me about vader,
>>>  I might have never realized it even existed.
>>> 
>>>  That could be part of one of the already existent FAQs
>>>  explaining how to select the btl.
>>> 
>>>  **
>>> 
>>>  Doubts (btl in OMPI 1.8):
>>> 
>>>  I still don't understand clearly the meaning and scope of vader
>>>  being a "default btl".
>>> 
>>>We mean that it has a higher priority than the other shared memory
>>>implementation, and so it will be used for intra-node messaging by
>>>default.
>>> 
>>>  Which is the scope of this default: intra-node btl only perhaps?
>>> 
>>>Yes - strictly intra-node
>>> 
>>>  Was there a default btl before vader, and which?
>>> 
>>>The "sm" btl was the default shared memory transport before vader
>>> 
>>>  Is vader the intra-node default only (i.e. replaces sm  by default),
>>> 
>>>Yes
>>> 
>>>  or does it somehow extend beyond node boundaries, and replaces (or
>>>  brings in) network btls (openib,tcp,etc) ?
>>> 
>>>Nope - just intra-node
>>> 
>>>  If I am running on several nodes, and want to use openib, not tcp,
>>>  and, say, use vader, what is the right syntax?
>>> 
>>>  * nothing (OMPI will figure it out ... but what if you have
>>>  IB,Ethernet,Myrinet,OpenGM, altogether?)
>>> 
>>>If you have higher-speed connections, we will pick the fastest for
>>>inter-node messaging as the "default" since we expect you would want the
>>>fastest possible transport.
>>> 
>>>  * -mca btl openib (and vader will come along automatically)
>>> 
>>>Among the ones you show, this would indeed be the likely choices (openib
>>>and vader)
>>> 
>>>  *

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-30 Thread Gus Correa 

Hi Nathan

Thank you very much for addressing this problem.

I read your notes on Jeff's blog about vader,
and that clarified many things that were obscure to me
when I first started this thread
whining that knem was not working in OMPI 1.8.3.
Thank you also for writing that blog post,
and for sending the link to it.
That was very helpful indeed.

As your closing comments on the blog post point out,
and your IMB benchmark graphs of pingpong/latency &
sendrecv/bandwidth show,
vader+xpmem outperforms the other combinations
of btl+memory_copy_mechanism of intra-node communication.

For the benefit of pedestrian OpenMPI users like me:

1) What is the status of xpmem in the Linux world at this point?
[Proprietary (SGI?) / open source, part of the Linux kernel (which),
part of standard distributions (which) ?]

2) Any recommendation for the values of the
various vader btl parameters?
[There are 12 of them in OMPI 1.8.3!
That is real challenge to get right.]

Which values did you use in your benchmarks?
Defaults?
Other?

In particular, is there an optimal value for the eager/rendevous 
threshold value? (btl_vader_eager_limit, default=4kB)
[The INRIA web site suggests 32kB for the sm+knem counterpart 
(btl_sm_eager_limit, default=4kB).]


3) Did I understand it right, that the upcoming OpenMPI 1.8.5
can be configured with more than one memory copy mechanism altogether
(e.g. --with-knem and --with-cma and --with-xpmem),
then select one of them at runtime with the 
btl_vader_single_copy_mechanism parameter?

Or must OMPI be configured with only one memory copy mechanism?

Many thanks,
Gus Correa


On 10/30/2014 05:44 PM, Nathan Hjelm wrote:

I want to close the loop on this issue. 1.8.5 will address it in several
ways:

  - knem support in btl/sm has been fixed. A sanity check was disabling
knem during component registration. I wrote the sanity check before
the 1.7 release and didn't intend this side-effect.

  - vader now supports xpmem, cma, and knem. The best available
single-copy mechanism will be used. If multiple single-copy
mechanisms are available you can select which one you want to use are
runtime.

More about the vader btl can be found here:
http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy/

-Nathan Hjelm
HPC-5, LANL

On Fri, Oct 17, 2014 at 01:02:23PM -0700, Ralph Castain wrote:

  On Oct 17, 2014, at 12:06 PM, Gus Correa  wrote:
  Hi Jeff

  Many thanks for looking into this and filing a bug report at 11:16PM!

  Thanks to Aurelien, Ralph and Nathan for their help and clarifications
  also.

  **

  Related suggestion:

  Add a note to the FAQ explaining that in OMPI 1.8
  the new (default) btl is vader (and what it is).

  It was a real surprise to me.
  If Aurelien Bouteiller didn't tell me about vader,
  I might have never realized it even existed.

  That could be part of one of the already existent FAQs
  explaining how to select the btl.

  **

  Doubts (btl in OMPI 1.8):

  I still don't understand clearly the meaning and scope of vader
  being a "default btl".

We mean that it has a higher priority than the other shared memory
implementation, and so it will be used for intra-node messaging by
default.

  Which is the scope of this default: intra-node btl only perhaps?

Yes - strictly intra-node

  Was there a default btl before vader, and which?

The "sm" btl was the default shared memory transport before vader

  Is vader the intra-node default only (i.e. replaces sm  by default),

Yes

  or does it somehow extend beyond node boundaries, and replaces (or
  brings in) network btls (openib,tcp,etc) ?

Nope - just intra-node

  If I am running on several nodes, and want to use openib, not tcp,
  and, say, use vader, what is the right syntax?

  * nothing (OMPI will figure it out ... but what if you have
  IB,Ethernet,Myrinet,OpenGM, altogether?)

If you have higher-speed connections, we will pick the fastest for
inter-node messaging as the "default" since we expect you would want the
fastest possible transport.

  * -mca btl openib (and vader will come along automatically)

Among the ones you show, this would indeed be the likely choices (openib
and vader)

  * -mca btl openib,self (and vader will come along automatically)

The "self" btl is *always* active as the loopback transport

  * -mca btl openib,self,vader (because vader is default only for 1-node
  jobs)
  * something else (or several alternatives)

  Whatever happened to the "self" btl in this new context?
  Gone? Still there?

  Many thanks,
  Gus Correa

  On 10/16/2014 11:16 PM, Jeff Squyres (jsquyres) wrote:

On Oct 16, 2014, at 1:35 PM, Gus Correa  wrote:

  and on the MCA parameter

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-30 Thread Nathan Hjelm 
I want to close the loop on this issue. 1.8.5 will address it in several
ways:

 - knem support in btl/sm has been fixed. A sanity check was disabling
   knem during component registration. I wrote the sanity check before
   the 1.7 release and didn't intend this side-effect.

 - vader now supports xpmem, cma, and knem. The best available
   single-copy mechanism will be used. If multiple single-copy
   mechanisms are available you can select which one you want to use are
   runtime.

More about the vader btl can be found here:
http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy/

-Nathan Hjelm
HPC-5, LANL

On Fri, Oct 17, 2014 at 01:02:23PM -0700, Ralph Castain wrote:
>  On Oct 17, 2014, at 12:06 PM, Gus Correa  wrote:
>  Hi Jeff
> 
>  Many thanks for looking into this and filing a bug report at 11:16PM!
> 
>  Thanks to Aurelien, Ralph and Nathan for their help and clarifications
>  also.
> 
>  **
> 
>  Related suggestion:
> 
>  Add a note to the FAQ explaining that in OMPI 1.8
>  the new (default) btl is vader (and what it is).
> 
>  It was a real surprise to me.
>  If Aurelien Bouteiller didn't tell me about vader,
>  I might have never realized it even existed.
> 
>  That could be part of one of the already existent FAQs
>  explaining how to select the btl.
> 
>  **
> 
>  Doubts (btl in OMPI 1.8):
> 
>  I still don't understand clearly the meaning and scope of vader
>  being a "default btl".
> 
>We mean that it has a higher priority than the other shared memory
>implementation, and so it will be used for intra-node messaging by
>default.
> 
>  Which is the scope of this default: intra-node btl only perhaps?
> 
>Yes - strictly intra-node
> 
>  Was there a default btl before vader, and which?
> 
>The "sm" btl was the default shared memory transport before vader
> 
>  Is vader the intra-node default only (i.e. replaces sm  by default),
> 
>Yes
> 
>  or does it somehow extend beyond node boundaries, and replaces (or
>  brings in) network btls (openib,tcp,etc) ?
> 
>Nope - just intra-node
> 
>  If I am running on several nodes, and want to use openib, not tcp,
>  and, say, use vader, what is the right syntax?
> 
>  * nothing (OMPI will figure it out ... but what if you have
>  IB,Ethernet,Myrinet,OpenGM, altogether?)
> 
>If you have higher-speed connections, we will pick the fastest for
>inter-node messaging as the "default" since we expect you would want the
>fastest possible transport.
> 
>  * -mca btl openib (and vader will come along automatically)
> 
>Among the ones you show, this would indeed be the likely choices (openib
>and vader)
> 
>  * -mca btl openib,self (and vader will come along automatically)
> 
>The "self" btl is *always* active as the loopback transport
> 
>  * -mca btl openib,self,vader (because vader is default only for 1-node
>  jobs)
>  * something else (or several alternatives)
> 
>  Whatever happened to the "self" btl in this new context?
>  Gone? Still there?
> 
>  Many thanks,
>  Gus Correa
> 
>  On 10/16/2014 11:16 PM, Jeff Squyres (jsquyres) wrote:
> 
>On Oct 16, 2014, at 1:35 PM, Gus Correa  wrote:
> 
>  and on the MCA parameter file:
> 
>  btl_sm_use_knem = 1
> 
>I think the logic enforcing this MCA param got broken when we revamped
>the MCA param system.  :-(
> 
>  I am scratching my head to understand why a parameter with such a
>  suggestive name ("btl_sm_have_knem_support"),
>  so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
>  somehow vanished from ompi_info in OMPI 1.8.3.
> 
>It looks like this MCA param was also dropped when we revamped the MCA
>system.  Doh!  :-(
> 
>There's some deep mojo going on that is somehow causing knem to not be
>used; I'm too tired to understand the logic right now.  I just opened
>https://github.com/open-mpi/ompi/issues/239 to track this issue --
>feel free to subscribe to the issue to get updates.
> 
>  ___
>  users mailing list
>  us...@open-mpi.org
>  Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>  Link to this
>  post: http://www.open-mpi.org/community/lists/users/2014/10/25532.php

> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/10/25534.php



pgp9iM_PC5QYR.pgp
Description: PGP signature

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-17 Thread Ralph Castain 

> On Oct 17, 2014, at 12:06 PM, Gus Correa  wrote:
> 
> Hi Jeff
> 
> Many thanks for looking into this and filing a bug report at 11:16PM!
> 
> Thanks to Aurelien, Ralph and Nathan for their help and clarifications
> also.
> 
> **
> 
> Related suggestion:
> 
> Add a note to the FAQ explaining that in OMPI 1.8
> the new (default) btl is vader (and what it is).
> 
> It was a real surprise to me.
> If Aurelien Bouteiller didn't tell me about vader,
> I might have never realized it even existed.
> 
> That could be part of one of the already existent FAQs
> explaining how to select the btl.
> 
> **
> 
> Doubts (btl in OMPI 1.8):
> 
> I still don't understand clearly the meaning and scope of vader
> being a "default btl”.

We mean that it has a higher priority than the other shared memory 
implementation, and so it will be used for intra-node messaging by default.

> Which is the scope of this default: intra-node btl only perhaps?

Yes - strictly intra-node

> Was there a default btl before vader, and which?

The “sm” btl was the default shared memory transport before vader

> Is vader the intra-node default only (i.e. replaces sm  by default),

Yes

> or does it somehow extend beyond node boundaries, and replaces (or brings in) 
> network btls (openib,tcp,etc) ?

Nope - just intra-node

> 
> If I am running on several nodes, and want to use openib, not tcp,
> and, say, use vader, what is the right syntax?
> 
> * nothing (OMPI will figure it out ... but what if you have 
> IB,Ethernet,Myrinet,OpenGM, altogether?)

If you have higher-speed connections, we will pick the fastest for inter-node 
messaging as the “default” since we expect you would want the fastest possible 
transport.

> * -mca btl openib (and vader will come along automatically)

Among the ones you show, this would indeed be the likely choices (openib and 
vader)

> * -mca btl openib,self (and vader will come along automatically)

The “self” btl is *always* active as the loopback transport

> * -mca btl openib,self,vader (because vader is default only for 1-node jobs)
> * something else (or several alternatives)
> 
> Whatever happened to the "self" btl in this new context?
> Gone? Still there?
> 
> Many thanks,
> Gus Correa
> 
> On 10/16/2014 11:16 PM, Jeff Squyres (jsquyres) wrote:
>> On Oct 16, 2014, at 1:35 PM, Gus Correa  wrote:
>> 
>>> and on the MCA parameter file:
>>> 
>>> btl_sm_use_knem = 1
>> 
>> I think the logic enforcing this MCA param got broken when we revamped the 
>> MCA param system.  :-(
>> 
>>> I am scratching my head to understand why a parameter with such a
>>> suggestive name ("btl_sm_have_knem_support"),
>>> so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
>>> somehow vanished from ompi_info in OMPI 1.8.3.
>> 
>> It looks like this MCA param was also dropped when we revamped the MCA 
>> system.  Doh!  :-(
>> 
>> There's some deep mojo going on that is somehow causing knem to not be used; 
>> I'm too tired to understand the logic right now.  I just opened 
>> https://github.com/open-mpi/ompi/issues/239 to track this issue -- feel free 
>> to subscribe to the issue to get updates.
>> 
> 
> ___
> users mailing list
> us...@open-mpi.org 
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users 
> 
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/10/25532.php 
>

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-17 Thread Gus Correa 

Hi Jeff

Many thanks for looking into this and filing a bug report at 11:16PM!

Thanks to Aurelien, Ralph and Nathan for their help and clarifications
also.

**

Related suggestion:

Add a note to the FAQ explaining that in OMPI 1.8
the new (default) btl is vader (and what it is).

It was a real surprise to me.
If Aurelien Bouteiller didn't tell me about vader,
I might have never realized it even existed.

That could be part of one of the already existent FAQs
explaining how to select the btl.

**

Doubts (btl in OMPI 1.8):

I still don't understand clearly the meaning and scope of vader
being a "default btl".
Which is the scope of this default: intra-node btl only perhaps?
Was there a default btl before vader, and which?
Is vader the intra-node default only (i.e. replaces sm  by default),
or does it somehow extend beyond node boundaries, and replaces (or 
brings in) network btls (openib,tcp,etc) ?


If I am running on several nodes, and want to use openib, not tcp,
and, say, use vader, what is the right syntax?

* nothing (OMPI will figure it out ... but what if you have 
IB,Ethernet,Myrinet,OpenGM, altogether?)

* -mca btl openib (and vader will come along automatically)
* -mca btl openib,self (and vader will come along automatically)
* -mca btl openib,self,vader (because vader is default only for 1-node jobs)
* something else (or several alternatives)

Whatever happened to the "self" btl in this new context?
Gone? Still there?

Many thanks,
Gus Correa

On 10/16/2014 11:16 PM, Jeff Squyres (jsquyres) wrote:

On Oct 16, 2014, at 1:35 PM, Gus Correa  wrote:


and on the MCA parameter file:

btl_sm_use_knem = 1


I think the logic enforcing this MCA param got broken when we revamped the MCA 
param system.  :-(


I am scratching my head to understand why a parameter with such a
suggestive name ("btl_sm_have_knem_support"),
so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
somehow vanished from ompi_info in OMPI 1.8.3.


It looks like this MCA param was also dropped when we revamped the MCA system.  
Doh!  :-(

There's some deep mojo going on that is somehow causing knem to not be used; 
I'm too tired to understand the logic right now.  I just opened 
https://github.com/open-mpi/ompi/issues/239 to track this issue -- feel free to 
subscribe to the issue to get updates.

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-17 Thread Jeff Squyres (jsquyres) 
On Oct 16, 2014, at 1:35 PM, Gus Correa  wrote:

> and on the MCA parameter file:
> 
> btl_sm_use_knem = 1

I think the logic enforcing this MCA param got broken when we revamped the MCA 
param system.  :-(

> I am scratching my head to understand why a parameter with such a
> suggestive name ("btl_sm_have_knem_support"),
> so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
> somehow vanished from ompi_info in OMPI 1.8.3.

It looks like this MCA param was also dropped when we revamped the MCA system.  
Doh!  :-(

There's some deep mojo going on that is somehow causing knem to not be used; 
I'm too tired to understand the logic right now.  I just opened 
https://github.com/open-mpi/ompi/issues/239 to track this issue -- feel free to 
subscribe to the issue to get updates.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa 

On 10/16/2014 07:32 PM, Jeff Squyres (jsquyres) wrote:

Gus --

Can you send the output of configure and your config.log?



Hi Jeff.

Sure.

This is for the OMPI 1.8.3 build with Intel compilers that I've been
using to compile and run IMB.
The config.log is attached.
The configure command and environment are
(is that what you meant by "the output of configure"?):

export CC=icc
export CXX=icpc
export FC=ifort
export CFLAGS='-msse2 -fp-model precise -Wall'
export CXXFLAGS=${CFLAGS}
export FCFLAGS='-msse2 -fp-model precise -warn all'

../configure \
--prefix=${MYINSTALLDIR} \
--with-tm=/opt/torque/4.2.5/gnu-4.4.7 \
--with-verbs=/usr \
--with-knem=/opt/knem-1.1.1 \
2>&1 | tee configure_${build_id}.log

Many thanks,
Gus



On Oct 16, 2014, at 4:24 PM, Gus Correa  wrote:


On 10/16/2014 05:38 PM, Nathan Hjelm wrote:

On Thu, Oct 16, 2014 at 05:27:54PM -0400, Gus Correa wrote:

Thank you, Aurelien!

Aha, "vader btl", that is new to me!
I tought Vader was that man dressed in black in Star Wars,
Obi-Wan Kenobi's nemesis.
That was a while ago, my kids were children,
and Alec Guiness younger than Harrison Ford is today.
Oh, how nostalgic code developers can get when it comes
to naming things ...

If I am using "vader", it is totally inadvertent.
There was no such a thing in Open MPI 1.6 and earlier.

Now that you mentioned, I can see lots of it in the 1.8.3
ompi_info output.
In addition, my stderr files show messages like this:

imb.e38352:[1,5]:[node13:16334] mca: bml: Not using sm btl to
[[59987,1],26] on node node13 because vader btl has higher exclusivity
(65536 > 65535)

So, you are right, "vader" is taking over and knocking off "sm" (and openib
and everybody else).
Darn Vader!
Probably knem is going down the tubes along with sm, right?


Depends. If there is a reason to continue supporting knem then vader
will be updated to support it. I don't currently see a reason to at this
time though (since sm continues to live for now).



Right now knem is not working in OMPI 1.8.3, even if I turn off vader,
and leave only sm,self,openib.
I just sent another email documenting that.


I was used to sm, openib, self and tcp BTLs.
I normally just do "btl = ^tcp" in the MCA parameters file,
to stick to sm, openib, and self.

That worked fine in 1.6.5 (and earlier), and knem worked
flawlessly there.
The same settings in 1.8.3 don't bring up the knem functionality.
So, this seems to be yet another change in 1.8.3 that I need to learn.

Can you or some other list subscriber elaborate a bit about
this 'vader' btl?
The Open MPI FAQ doesn't have anthing about it.
What is it after all?
Does it play the same role as "sm", i.e., an intra-node btl?
Considering the name, is "vader" good or bad?
Or better: In which circumstances is "vader" good and when is it bad?


Vader is a btl I originally wrote to support Cray's XPMEM shared memory
interface. It was designed to be cleaner than btl/sm have better small
message latency, bandwidth, and message rates. Because its latency is so
much better than sm I removed the XPMEM requirement and added CMA
support.



I presume this requires kernel 3.X, as Aurelien pointed out.
As a matter of policy, and to keep your user base broad,
I would suggest to keep a generous
range of backwards compatible support built into OMPI.
This would be sm, knem, etc, which I suppose can coexist with vader, or not?
I can't speak for others but we run production codes in
standard Linux distributions (Centos 6.X, 5.X) whith 2.6.Y kernels.
I suppose other people have similar situations.


Should I give in to the dark side of the force and keep "vader"
turned on, or should I just do something like
"btl = ^tcp,^vader" ?


You can turn off vader if you want to use knem. I would run some tests
to see if there is much of a difference between sm/knem and vader
though. I don't have any systems that have knem installed so I haven't
been able to run these tests myself. I would primarily focus on the
memory usage and the bandwidth.

-Nathan


Please, see my last email.
Turning off vader and sm on, still doesn't make knem work,
unless I made some big mistake along the way.
I would love to use 1.8.3 in production,
as long as sm+knem support works, hence it it would be
great if somebody points out any mistake that I may have made.

Also, for large messages, IMB with 1.6.5+sm+knem gives
me ~30% speedups w.r.t. 1.8.3+sm+(broken)-knem or w.r.t. 1.8.3+vader,
although admittedly due to our 2.6 kernel, no CMA, etc,
the environment is not favorable to vader to begin with.
[And yet another good reason to fix/keep sm+knem in OMPI 1.8.]

Thank you,
Gus Correa





___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/10/25516.php



___
users mailing list
us...@open-mpi.org
Subscription:

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa 

Hi Ralph

I have clusters with CentOS 6.4, 6.5, and 5.5.

OK, completing my table (ran on CentOS 6.4):

#bytes #repetitions  t[usec]   Mbytes/sec
262144  16048.04  5203.93 :OMPI 1.6.5+knem
262144  16063.72  3923.30 :OMPI 1.8.3+vader
262144  16067.00  3731.34 :OMPI 1.8.3+sm+knem_broken

Indeed, as you said, in 1.8.3 vader is faster than sm (lines 2 an 3).
However, 1.6.5+knem still beats the crap on both (line 1).
Anyway, as others have said, vader needs the CMA thing,
kernel 3.X, etc, which I don't have, so comparing knem with
vader-less-the-support-it-requires is bogus.

Nevertheless, I would guess I am not the only one to live
in a kernel 2.6.Z world, hence still need knem support.

It would be great if the knem functionality could be restored to
OMPI 1.8.

Many thanks,
Gus

On 10/16/2014 07:13 PM, Ralph Castain wrote:

You probably have this somewhere below, but what OS are you running?

I have CentOS6, and vader works fine for me and is much faster than the
sm btl.


I can certainly ask to see if someone has time to fix the knem support -

if they do, we would definitely include the fix in the 1.8 series.



On Oct 16, 2014, at 4:06 PM, Gus Correa  wrote:


Hi All

Back to the original issue of knem in Open MPI 1.8.3.
It really seems to be broken.

I launched the Intel MPI benchmarks (IMB) job both with
'-mca btl ^vader,tcp', and with '-mca btl sm,self,openib'.
Both syntaxes seem to have turned off vader (along with tcp),
as shown in stderr by messages like this
(I also used -mca btl_base_verbose 30):

[1,11]:[node26:13439] mca: bml: Using sm btl to [[39251,1],0] on node 
node26

*However*, in both cases /dev/knem continues to *show only zeros*.

My conclusion is that the knem seems not to be working
at all in OMPI 1.8.3.

That is a real pity, because without knem performance really suffers.
I took a quick look at the Intel MPI benchmarks output
using OMPI 1.6.5 with knem, and OMPI 1.8.5 where knem doesn't work (despite my 
attempts to make it work).
The older OMPI with knem shows very good speedups.
For instance, ping-pong on two processors, message size 256kB,
OMPI 1.6.5+knem has a ~32% speeedup w.r.t. OMPI 1.8.3.

#bytes #repetitions  t[usec]   Mbytes/sec
262144  16048.04  5203.93 (OMPI 1.6.5 + knem)
262144  16063.72  3923.30 (OMPI 1.8.3, broken knem)

Numbers like these don't give me any incentive to upgrade
our production codes to OMPI 1.8.
Will this be fixed in the next Open MPI 1.8 release?

Thank you,
Gus Correa

PS - Many thanks to Aurelien Boutelier for pointing out the existence
of the vader btl.  Without his tip I would still be in the dark side.

On 10/16/2014 05:46 PM, Gus Correa wrote:


On 10/16/2014 05:28 PM, Nathan Hjelm wrote:

And it doesn't support knem at this time. Probably never will because of
the existence of CMA.

-Nathan



Thanks, Nathan

But for the benefit of mere mortals like me
who don't share the dark or the bright side of the force,
and just need to keep their MPI applications running in production mode,
hopefully with Open MPI 1.8,
can somebody explain more clearly what "vader" is about?

Thank you,
Gus Correa



On Thu, Oct 16, 2014 at 01:49:09PM -0700, Ralph Castain wrote:

FWIW: vader is the default in 1.8

On Oct 16, 2014, at 1:40 PM, Aurélien Bouteiller
 wrote:


Are you sure you are not using the vader BTL ?

Setting mca_btl_base_verbose and/or sm_verbose should spit out some
knem initialization info.

The CMA linux system (that ships with most 3.1x linux kernels) has
similar features, and is also supported in sm.

Aurelien
--
  ~~~ Aurélien Bouteiller, Ph.D. ~~~
 ~ Research Scientist @ ICL ~
The University of Tennessee, Innovative Computing Laboratory
1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
https://icl.cs.utk.edu/~bouteill/




Le 16 oct. 2014 à 16:35, Gus Correa  a écrit :


Dear Open MPI developers

Well, I just can't keep my promises for too long ...
So, here I am pestering you again, although this time
it is not a request for more documentation.
Hopefully it is something more legit.

I am having trouble using knem with Open MPI 1.8.3,
and need your help.

I configured Open MPI 1.8.3 with knem.
I had done the same with some builds of Open MPI 1.6.5 before.

When I build and launch the Intel MPI benchmarks (IMB)
with Open MPI 1.6.5,
'cat /dev/knem'
starts showing non-zero-and-growing statistics right away.

However, when I build and launch IMB with Open MPI 1.8.3,
/dev/knem shows only zeros,
no statistics growing, nothing.
Knem just seems to be completely asleep.

So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
at least not for me.

***

The runtime environment related to knem is setup the
same way on both OPMI releases.
I tried setting it up both on the command line:

-mca

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Jeff Squyres (jsquyres) 
Gus --

Can you send the output of configure and your config.log?


On Oct 16, 2014, at 4:24 PM, Gus Correa  wrote:

> On 10/16/2014 05:38 PM, Nathan Hjelm wrote:
>> On Thu, Oct 16, 2014 at 05:27:54PM -0400, Gus Correa wrote:
>>> Thank you, Aurelien!
>>> 
>>> Aha, "vader btl", that is new to me!
>>> I tought Vader was that man dressed in black in Star Wars,
>>> Obi-Wan Kenobi's nemesis.
>>> That was a while ago, my kids were children,
>>> and Alec Guiness younger than Harrison Ford is today.
>>> Oh, how nostalgic code developers can get when it comes
>>> to naming things ...
>>> 
>>> If I am using "vader", it is totally inadvertent.
>>> There was no such a thing in Open MPI 1.6 and earlier.
>>> 
>>> Now that you mentioned, I can see lots of it in the 1.8.3
>>> ompi_info output.
>>> In addition, my stderr files show messages like this:
>>> 
>>> imb.e38352:[1,5]:[node13:16334] mca: bml: Not using sm btl to
>>> [[59987,1],26] on node node13 because vader btl has higher exclusivity
>>> (65536 > 65535)
>>> 
>>> So, you are right, "vader" is taking over and knocking off "sm" (and openib
>>> and everybody else).
>>> Darn Vader!
>>> Probably knem is going down the tubes along with sm, right?
>> 
>> Depends. If there is a reason to continue supporting knem then vader
>> will be updated to support it. I don't currently see a reason to at this
>> time though (since sm continues to live for now).
>> 
> 
> Right now knem is not working in OMPI 1.8.3, even if I turn off vader,
> and leave only sm,self,openib.
> I just sent another email documenting that.
> 
>>> I was used to sm, openib, self and tcp BTLs.
>>> I normally just do "btl = ^tcp" in the MCA parameters file,
>>> to stick to sm, openib, and self.
>>> 
>>> That worked fine in 1.6.5 (and earlier), and knem worked
>>> flawlessly there.
>>> The same settings in 1.8.3 don't bring up the knem functionality.
>>> So, this seems to be yet another change in 1.8.3 that I need to learn.
>>> 
>>> Can you or some other list subscriber elaborate a bit about
>>> this 'vader' btl?
>>> The Open MPI FAQ doesn't have anthing about it.
>>> What is it after all?
>>> Does it play the same role as "sm", i.e., an intra-node btl?
>>> Considering the name, is "vader" good or bad?
>>> Or better: In which circumstances is "vader" good and when is it bad?
>> 
>> Vader is a btl I originally wrote to support Cray's XPMEM shared memory
>> interface. It was designed to be cleaner than btl/sm have better small
>> message latency, bandwidth, and message rates. Because its latency is so
>> much better than sm I removed the XPMEM requirement and added CMA
>> support.
>> 
> 
> I presume this requires kernel 3.X, as Aurelien pointed out.
> As a matter of policy, and to keep your user base broad,
> I would suggest to keep a generous
> range of backwards compatible support built into OMPI.
> This would be sm, knem, etc, which I suppose can coexist with vader, or not?
> I can't speak for others but we run production codes in
> standard Linux distributions (Centos 6.X, 5.X) whith 2.6.Y kernels.
> I suppose other people have similar situations.
> 
>>> Should I give in to the dark side of the force and keep "vader"
>>> turned on, or should I just do something like
>>> "btl = ^tcp,^vader" ?
>> 
>> You can turn off vader if you want to use knem. I would run some tests
>> to see if there is much of a difference between sm/knem and vader
>> though. I don't have any systems that have knem installed so I haven't
>> been able to run these tests myself. I would primarily focus on the
>> memory usage and the bandwidth.
> >
> > -Nathan
> 
> Please, see my last email.
> Turning off vader and sm on, still doesn't make knem work,
> unless I made some big mistake along the way.
> I would love to use 1.8.3 in production,
> as long as sm+knem support works, hence it it would be
> great if somebody points out any mistake that I may have made.
> 
> Also, for large messages, IMB with 1.6.5+sm+knem gives
> me ~30% speedups w.r.t. 1.8.3+sm+(broken)-knem or w.r.t. 1.8.3+vader,
> although admittedly due to our 2.6 kernel, no CMA, etc,
> the environment is not favorable to vader to begin with.
> [And yet another good reason to fix/keep sm+knem in OMPI 1.8.]
> 
> Thank you,
> Gus Correa
> 
>> 
>> 
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2014/10/25516.php
>> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/10/25521.php


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa 

On 10/16/2014 05:38 PM, Nathan Hjelm wrote:

On Thu, Oct 16, 2014 at 05:27:54PM -0400, Gus Correa wrote:

Thank you, Aurelien!

Aha, "vader btl", that is new to me!
I tought Vader was that man dressed in black in Star Wars,
Obi-Wan Kenobi's nemesis.
That was a while ago, my kids were children,
and Alec Guiness younger than Harrison Ford is today.
Oh, how nostalgic code developers can get when it comes
to naming things ...

If I am using "vader", it is totally inadvertent.
There was no such a thing in Open MPI 1.6 and earlier.

Now that you mentioned, I can see lots of it in the 1.8.3
ompi_info output.
In addition, my stderr files show messages like this:

imb.e38352:[1,5]:[node13:16334] mca: bml: Not using sm btl to
[[59987,1],26] on node node13 because vader btl has higher exclusivity
(65536 > 65535)

So, you are right, "vader" is taking over and knocking off "sm" (and openib
and everybody else).
Darn Vader!
Probably knem is going down the tubes along with sm, right?


Depends. If there is a reason to continue supporting knem then vader
will be updated to support it. I don't currently see a reason to at this
time though (since sm continues to live for now).



Right now knem is not working in OMPI 1.8.3, even if I turn off vader,
and leave only sm,self,openib.
I just sent another email documenting that.


I was used to sm, openib, self and tcp BTLs.
I normally just do "btl = ^tcp" in the MCA parameters file,
to stick to sm, openib, and self.

That worked fine in 1.6.5 (and earlier), and knem worked
flawlessly there.
The same settings in 1.8.3 don't bring up the knem functionality.
So, this seems to be yet another change in 1.8.3 that I need to learn.

Can you or some other list subscriber elaborate a bit about
this 'vader' btl?
The Open MPI FAQ doesn't have anthing about it.
What is it after all?
Does it play the same role as "sm", i.e., an intra-node btl?
Considering the name, is "vader" good or bad?
Or better: In which circumstances is "vader" good and when is it bad?


Vader is a btl I originally wrote to support Cray's XPMEM shared memory
interface. It was designed to be cleaner than btl/sm have better small
message latency, bandwidth, and message rates. Because its latency is so
much better than sm I removed the XPMEM requirement and added CMA
support.



I presume this requires kernel 3.X, as Aurelien pointed out.
As a matter of policy, and to keep your user base broad,
I would suggest to keep a generous
range of backwards compatible support built into OMPI.
This would be sm, knem, etc, which I suppose can coexist with vader, or not?
I can't speak for others but we run production codes in
standard Linux distributions (Centos 6.X, 5.X) whith 2.6.Y kernels.
I suppose other people have similar situations.


Should I give in to the dark side of the force and keep "vader"
turned on, or should I just do something like
"btl = ^tcp,^vader" ?


You can turn off vader if you want to use knem. I would run some tests
to see if there is much of a difference between sm/knem and vader
though. I don't have any systems that have knem installed so I haven't
been able to run these tests myself. I would primarily focus on the
memory usage and the bandwidth.

>
> -Nathan

Please, see my last email.
Turning off vader and sm on, still doesn't make knem work,
unless I made some big mistake along the way.
I would love to use 1.8.3 in production,
as long as sm+knem support works, hence it it would be
great if somebody points out any mistake that I may have made.

Also, for large messages, IMB with 1.6.5+sm+knem gives
me ~30% speedups w.r.t. 1.8.3+sm+(broken)-knem or w.r.t. 1.8.3+vader,
although admittedly due to our 2.6 kernel, no CMA, etc,
the environment is not favorable to vader to begin with.
[And yet another good reason to fix/keep sm+knem in OMPI 1.8.]

Thank you,
Gus Correa





___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/10/25516.php

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Ralph Castain 
You probably have this somewhere below, but what OS are you running? I have 
CentOS6, and vader works fine for me and is much faster than the sm btl.

I can certainly ask to see if someone has time to fix the knem support - if 
they do, we would definitely include the fix in the 1.8 series.


On Oct 16, 2014, at 4:06 PM, Gus Correa  wrote:

> Hi All
> 
> Back to the original issue of knem in Open MPI 1.8.3.
> It really seems to be broken.
> 
> I launched the Intel MPI benchmarks (IMB) job both with
> '-mca btl ^vader,tcp', and with '-mca btl sm,self,openib'.
> Both syntaxes seem to have turned off vader (along with tcp),
> as shown in stderr by messages like this
> (I also used -mca btl_base_verbose 30):
> 
> [1,11]:[node26:13439] mca: bml: Using sm btl to [[39251,1],0] on 
> node node26
> 
> *However*, in both cases /dev/knem continues to *show only zeros*.
> 
> My conclusion is that the knem seems not to be working
> at all in OMPI 1.8.3.
> 
> That is a real pity, because without knem performance really suffers.
> I took a quick look at the Intel MPI benchmarks output
> using OMPI 1.6.5 with knem, and OMPI 1.8.5 where knem doesn't work (despite 
> my attempts to make it work).
> The older OMPI with knem shows very good speedups.
> For instance, ping-pong on two processors, message size 256kB,
> OMPI 1.6.5+knem has a ~32% speeedup w.r.t. OMPI 1.8.3.
> 
> #bytes #repetitions  t[usec]   Mbytes/sec
> 262144  16048.04  5203.93 (OMPI 1.6.5 + knem)
> 262144  16063.72  3923.30 (OMPI 1.8.3, broken knem)
> 
> Numbers like these don't give me any incentive to upgrade
> our production codes to OMPI 1.8.
> Will this be fixed in the next Open MPI 1.8 release?
> 
> Thank you,
> Gus Correa
> 
> PS - Many thanks to Aurelien Boutelier for pointing out the existence
> of the vader btl.  Without his tip I would still be in the dark side.
> 
> On 10/16/2014 05:46 PM, Gus Correa wrote:
>> 
>> On 10/16/2014 05:28 PM, Nathan Hjelm wrote:
>>> And it doesn't support knem at this time. Probably never will because of
>>> the existence of CMA.
>>> 
>>> -Nathan
>>> 
>> 
>> Thanks, Nathan
>> 
>> But for the benefit of mere mortals like me
>> who don't share the dark or the bright side of the force,
>> and just need to keep their MPI applications running in production mode,
>> hopefully with Open MPI 1.8,
>> can somebody explain more clearly what "vader" is about?
>> 
>> Thank you,
>> Gus Correa
>> 
>> 
>>> On Thu, Oct 16, 2014 at 01:49:09PM -0700, Ralph Castain wrote:
 FWIW: vader is the default in 1.8
 
 On Oct 16, 2014, at 1:40 PM, Aurélien Bouteiller
  wrote:
 
> Are you sure you are not using the vader BTL ?
> 
> Setting mca_btl_base_verbose and/or sm_verbose should spit out some
> knem initialization info.
> 
> The CMA linux system (that ships with most 3.1x linux kernels) has
> similar features, and is also supported in sm.
> 
> Aurelien
> --
>  ~~~ Aurélien Bouteiller, Ph.D. ~~~
> ~ Research Scientist @ ICL ~
> The University of Tennessee, Innovative Computing Laboratory
> 1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
> tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
> https://icl.cs.utk.edu/~bouteill/
> 
> 
> 
> 
> Le 16 oct. 2014 à 16:35, Gus Correa  a écrit :
> 
>> Dear Open MPI developers
>> 
>> Well, I just can't keep my promises for too long ...
>> So, here I am pestering you again, although this time
>> it is not a request for more documentation.
>> Hopefully it is something more legit.
>> 
>> I am having trouble using knem with Open MPI 1.8.3,
>> and need your help.
>> 
>> I configured Open MPI 1.8.3 with knem.
>> I had done the same with some builds of Open MPI 1.6.5 before.
>> 
>> When I build and launch the Intel MPI benchmarks (IMB)
>> with Open MPI 1.6.5,
>> 'cat /dev/knem'
>> starts showing non-zero-and-growing statistics right away.
>> 
>> However, when I build and launch IMB with Open MPI 1.8.3,
>> /dev/knem shows only zeros,
>> no statistics growing, nothing.
>> Knem just seems to be completely asleep.
>> 
>> So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
>> at least not for me.
>> 
>> ***
>> 
>> The runtime environment related to knem is setup the
>> same way on both OPMI releases.
>> I tried setting it up both on the command line:
>> 
>> -mca btl_sm_eager_limit 32768 -mca btl_sm_knem_dma_min 1048576
>> 
>> and on the MCA parameter file:
>> 
>> btl_sm_use_knem = 1
>> btl_sm_eager_limit = 32768
>> btl_sm_knem_dma_min = 1048576
>> 
>> and the behavior is the same (i.e., knem is active in 1.6.5,
>> but doesn't seem to be used by 1.8.3, as indicated by the
>> /dev/knem

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa 

Hi All

Back to the original issue of knem in Open MPI 1.8.3.
It really seems to be broken.

I launched the Intel MPI benchmarks (IMB) job both with
'-mca btl ^vader,tcp', and with '-mca btl sm,self,openib'.
Both syntaxes seem to have turned off vader (along with tcp),
as shown in stderr by messages like this
(I also used -mca btl_base_verbose 30):

[1,11]:[node26:13439] mca: bml: Using sm btl to [[39251,1],0] 
on node node26


*However*, in both cases /dev/knem continues to *show only zeros*.

My conclusion is that the knem seems not to be working
at all in OMPI 1.8.3.

That is a real pity, because without knem performance really suffers.
I took a quick look at the Intel MPI benchmarks output
using OMPI 1.6.5 with knem, and OMPI 1.8.5 where knem doesn't work 
(despite my attempts to make it work).

The older OMPI with knem shows very good speedups.
For instance, ping-pong on two processors, message size 256kB,
OMPI 1.6.5+knem has a ~32% speeedup w.r.t. OMPI 1.8.3.

#bytes #repetitions  t[usec]   Mbytes/sec
262144  16048.04  5203.93 (OMPI 1.6.5 + knem)
262144  16063.72  3923.30 (OMPI 1.8.3, broken knem)

Numbers like these don't give me any incentive to upgrade
our production codes to OMPI 1.8.
Will this be fixed in the next Open MPI 1.8 release?

Thank you,
Gus Correa

PS - Many thanks to Aurelien Boutelier for pointing out the existence
of the vader btl.  Without his tip I would still be in the dark side.

On 10/16/2014 05:46 PM, Gus Correa wrote:


On 10/16/2014 05:28 PM, Nathan Hjelm wrote:

And it doesn't support knem at this time. Probably never will because of
the existence of CMA.

-Nathan



Thanks, Nathan

But for the benefit of mere mortals like me
who don't share the dark or the bright side of the force,
and just need to keep their MPI applications running in production mode,
hopefully with Open MPI 1.8,
can somebody explain more clearly what "vader" is about?

Thank you,
Gus Correa



On Thu, Oct 16, 2014 at 01:49:09PM -0700, Ralph Castain wrote:

FWIW: vader is the default in 1.8

On Oct 16, 2014, at 1:40 PM, Aurélien Bouteiller
 wrote:


Are you sure you are not using the vader BTL ?

Setting mca_btl_base_verbose and/or sm_verbose should spit out some
knem initialization info.

The CMA linux system (that ships with most 3.1x linux kernels) has
similar features, and is also supported in sm.

Aurelien
--
  ~~~ Aurélien Bouteiller, Ph.D. ~~~
 ~ Research Scientist @ ICL ~
The University of Tennessee, Innovative Computing Laboratory
1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
https://icl.cs.utk.edu/~bouteill/




Le 16 oct. 2014 à 16:35, Gus Correa  a écrit :


Dear Open MPI developers

Well, I just can't keep my promises for too long ...
So, here I am pestering you again, although this time
it is not a request for more documentation.
Hopefully it is something more legit.

I am having trouble using knem with Open MPI 1.8.3,
and need your help.

I configured Open MPI 1.8.3 with knem.
I had done the same with some builds of Open MPI 1.6.5 before.

When I build and launch the Intel MPI benchmarks (IMB)
with Open MPI 1.6.5,
'cat /dev/knem'
starts showing non-zero-and-growing statistics right away.

However, when I build and launch IMB with Open MPI 1.8.3,
/dev/knem shows only zeros,
no statistics growing, nothing.
Knem just seems to be completely asleep.

So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
at least not for me.

***

The runtime environment related to knem is setup the
same way on both OPMI releases.
I tried setting it up both on the command line:

-mca btl_sm_eager_limit 32768 -mca btl_sm_knem_dma_min 1048576

and on the MCA parameter file:

btl_sm_use_knem = 1
btl_sm_eager_limit = 32768
btl_sm_knem_dma_min = 1048576

and the behavior is the same (i.e., knem is active in 1.6.5,
but doesn't seem to be used by 1.8.3, as indicated by the
/dev/knem statistics.)

***

When I 'grep -i knem config.log', both 1.6.5 and 1.8.3 builds show:

#define OMPI_BTL_SM_HAVE_KNEM 1

suggesting that both configurations picked up knem correctly.

On the other hand, when I do 'ompi_info --all --all |grep knem',
OMPI 1.6.5 shows "btl_sm_have_knem_support":

'MCA btl: information "btl_sm_have_knem_support" (value: <1>, data
source: default value)  Whether this component supports the knem
Linux kernel module or not'

By contrast, in OMPI 1.8.3 ompi_info doesn't show this particular
item ("btl_sm_have_knem_support"),
although the *other* 'btl sm knem' items are there,
namely "btl_sm_use_knem","btl_sm_knem_dma_min",
"btl_sm_knem_max_simultaneous".

I am scratching my head to understand why a parameter with such a
suggestive name ("btl_sm_have_knem_support"),
so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
somehow vanished from ompi_info in OMPI 1.8.3.

***

Questions:

- Am I doing something totally wrong,
perhaps with

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa 


On 10/16/2014 05:28 PM, Nathan Hjelm wrote:

And it doesn't support knem at this time. Probably never will because of
the existence of CMA.

-Nathan



Thanks, Nathan

But for the benefit of mere mortals like me
who don't share the dark or the bright side of the force,
and just need to keep their MPI applications running in production mode,
hopefully with Open MPI 1.8,
can somebody explain more clearly what "vader" is about?

Thank you,
Gus Correa



On Thu, Oct 16, 2014 at 01:49:09PM -0700, Ralph Castain wrote:

FWIW: vader is the default in 1.8

On Oct 16, 2014, at 1:40 PM, Aurélien Bouteiller  wrote:


Are you sure you are not using the vader BTL ?

Setting mca_btl_base_verbose and/or sm_verbose should spit out some knem 
initialization info.

The CMA linux system (that ships with most 3.1x linux kernels) has similar 
features, and is also supported in sm.

Aurelien
--
  ~~~ Aurélien Bouteiller, Ph.D. ~~~
 ~ Research Scientist @ ICL ~
The University of Tennessee, Innovative Computing Laboratory
1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
https://icl.cs.utk.edu/~bouteill/




Le 16 oct. 2014 à 16:35, Gus Correa  a écrit :


Dear Open MPI developers

Well, I just can't keep my promises for too long ...
So, here I am pestering you again, although this time
it is not a request for more documentation.
Hopefully it is something more legit.

I am having trouble using knem with Open MPI 1.8.3,
and need your help.

I configured Open MPI 1.8.3 with knem.
I had done the same with some builds of Open MPI 1.6.5 before.

When I build and launch the Intel MPI benchmarks (IMB)
with Open MPI 1.6.5,
'cat /dev/knem'
starts showing non-zero-and-growing statistics right away.

However, when I build and launch IMB with Open MPI 1.8.3,
/dev/knem shows only zeros,
no statistics growing, nothing.
Knem just seems to be completely asleep.

So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
at least not for me.

***

The runtime environment related to knem is setup the
same way on both OPMI releases.
I tried setting it up both on the command line:

-mca btl_sm_eager_limit 32768 -mca btl_sm_knem_dma_min 1048576

and on the MCA parameter file:

btl_sm_use_knem = 1
btl_sm_eager_limit = 32768
btl_sm_knem_dma_min = 1048576

and the behavior is the same (i.e., knem is active in 1.6.5,
but doesn't seem to be used by 1.8.3, as indicated by the
/dev/knem statistics.)

***

When I 'grep -i knem config.log', both 1.6.5 and 1.8.3 builds show:

#define OMPI_BTL_SM_HAVE_KNEM 1

suggesting that both configurations picked up knem correctly.

On the other hand, when I do 'ompi_info --all --all |grep knem',
OMPI 1.6.5 shows "btl_sm_have_knem_support":

'MCA btl: information "btl_sm_have_knem_support" (value: <1>, data source: 
default value)  Whether this component supports the knem Linux kernel module or not'

By contrast, in OMPI 1.8.3 ompi_info doesn't show this particular item 
("btl_sm_have_knem_support"),
although the *other* 'btl sm knem' items are there,
namely "btl_sm_use_knem","btl_sm_knem_dma_min", "btl_sm_knem_max_simultaneous".

I am scratching my head to understand why a parameter with such a
suggestive name ("btl_sm_have_knem_support"),
so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
somehow vanished from ompi_info in OMPI 1.8.3.

***

Questions:

- Am I doing something totally wrong,
perhaps with the knem runtime environment?

- Was knem somehow phased out in 1.8.3?

- Could there be a bad interaction with other runtime parameters that
somehow is knocking out knem in 1.8.3?
(FYI, besides knem, I'm just excluding the tcp btl, binding to core, and 
reporting the bindings, which is exactly what I do on 1.6.5,
although the runtime parameter syntax has changed.)

- Is knem inadvertently not being activated at runtime in OMPI 1.8.3?
(i.e. a bug)

- Is there a way to increase verbosity to detect if knem is being
used by OMPI?
That would certainly help to check what is going on.
I tried '-mca btl_base_verbose 30' but there was no trace of knem
in sderr/stdout of either 1.6.5 or 1.8.3.
So, the evidence I have that knem is
active in 1.6.5 but not in 1.8.3 comes only from the statistics in
/dev/knem.

***


Thank you,
Gus Correa

***

PS - As an aside, I also have some questions on the knem setup,
which I mostly copied from the knem web site
(hopefully Brice Goglin is listening ...):

- Is 32768 in 'btl_sm_eager_limit 32768' a good number,
or should it be larger/smaller/something else?
[OK, I know I should benchmark it, but exploring the whole parameter
space takes long, so why not asking? ]

- Is it worth using 'btl_sm_knem_dma_min 1048576'?
[I think I read somewhere that this dma engine offload
is an Intel thing, not AMD.]

- How about btl_sm_knem_max_simultaneous?
That one is not mentioned in the knem web site.
Should I leave it default to zero or set it to 1? 2? 4? Something else?


Thanks

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa 

On 10/16/2014 04:49 PM, Ralph Castain wrote:
> FWIW: vader is the default in 1.8

Yes, Ralph, thank you, I just noticed it in my job's stderr,
after Aurelien pointed out that new "vader" thing existed.
What a quick promotion: from inexistent to default btl!

But what is "vader" after all?
Any pointers, links, ... eh ... documentation (oops, I said I would not 
ask for it ...)


There is nothing about "vader" in the FAQ.
How does it affect the other btl?
Is it some kind of "wrapper btl" that decides which of the lower
level btls to use (sm, openib, etc)?
How does it affect knem?
What are vader's pros/cons w.r.t. using the other btls?
In which conditions is it good or bad to use it vs. the other btls?
What do I gain/lose if I do "btl = sm,self,openib"
(which presumably will knock off tcp and "vader'),
or maybe "btl=^tcp,^vader" ?

Many thanks,
Gus Correa

> On Oct 16, 2014, at 1:40 PM, Aurélien Bouteiller 
 wrote:



Are you sure you are not using the vader BTL ?

Setting mca_btl_base_verbose and/or sm_verbose should spit out some knem 
initialization info.

The CMA linux system (that ships with most 3.1x linux kernels) has similar 
features, and is also supported in sm.

Aurelien
--
  ~~~ Aurélien Bouteiller, Ph.D. ~~~
 ~ Research Scientist @ ICL ~
The University of Tennessee, Innovative Computing Laboratory
1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
https://icl.cs.utk.edu/~bouteill/




Le 16 oct. 2014 à 16:35, Gus Correa  a écrit :


Dear Open MPI developers

Well, I just can't keep my promises for too long ...
So, here I am pestering you again, although this time
it is not a request for more documentation.
Hopefully it is something more legit.

I am having trouble using knem with Open MPI 1.8.3,
and need your help.

I configured Open MPI 1.8.3 with knem.
I had done the same with some builds of Open MPI 1.6.5 before.

When I build and launch the Intel MPI benchmarks (IMB)
with Open MPI 1.6.5,
'cat /dev/knem'
starts showing non-zero-and-growing statistics right away.

However, when I build and launch IMB with Open MPI 1.8.3,
/dev/knem shows only zeros,
no statistics growing, nothing.
Knem just seems to be completely asleep.

So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
at least not for me.

***

The runtime environment related to knem is setup the
same way on both OPMI releases.
I tried setting it up both on the command line:

-mca btl_sm_eager_limit 32768 -mca btl_sm_knem_dma_min 1048576

and on the MCA parameter file:

btl_sm_use_knem = 1
btl_sm_eager_limit = 32768
btl_sm_knem_dma_min = 1048576

and the behavior is the same (i.e., knem is active in 1.6.5,
but doesn't seem to be used by 1.8.3, as indicated by the
/dev/knem statistics.)

***

When I 'grep -i knem config.log', both 1.6.5 and 1.8.3 builds show:

#define OMPI_BTL_SM_HAVE_KNEM 1

suggesting that both configurations picked up knem correctly.

On the other hand, when I do 'ompi_info --all --all |grep knem',
OMPI 1.6.5 shows "btl_sm_have_knem_support":

'MCA btl: information "btl_sm_have_knem_support" (value: <1>, data source: 
default value)  Whether this component supports the knem Linux kernel module or not'

By contrast, in OMPI 1.8.3 ompi_info doesn't show this particular item 
("btl_sm_have_knem_support"),
although the *other* 'btl sm knem' items are there,
namely "btl_sm_use_knem","btl_sm_knem_dma_min", "btl_sm_knem_max_simultaneous".

I am scratching my head to understand why a parameter with such a
suggestive name ("btl_sm_have_knem_support"),
so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
somehow vanished from ompi_info in OMPI 1.8.3.

***

Questions:

- Am I doing something totally wrong,
perhaps with the knem runtime environment?

- Was knem somehow phased out in 1.8.3?

- Could there be a bad interaction with other runtime parameters that
somehow is knocking out knem in 1.8.3?
(FYI, besides knem, I'm just excluding the tcp btl, binding to core, and 
reporting the bindings, which is exactly what I do on 1.6.5,
although the runtime parameter syntax has changed.)

- Is knem inadvertently not being activated at runtime in OMPI 1.8.3?
(i.e. a bug)

- Is there a way to increase verbosity to detect if knem is being
used by OMPI?
That would certainly help to check what is going on.
I tried '-mca btl_base_verbose 30' but there was no trace of knem
in sderr/stdout of either 1.6.5 or 1.8.3.
So, the evidence I have that knem is
active in 1.6.5 but not in 1.8.3 comes only from the statistics in
/dev/knem.

***


Thank you,
Gus Correa

***

PS - As an aside, I also have some questions on the knem setup,
which I mostly copied from the knem web site
(hopefully Brice Goglin is listening ...):

- Is 32768 in 'btl_sm_eager_limit 32768' a good number,
or should it be larger/smaller/something else?
[OK, I know I should benchmark it, but exploring the whole parameter
space takes long, so why not

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Nathan Hjelm 
On Thu, Oct 16, 2014 at 05:27:54PM -0400, Gus Correa wrote:
> Thank you, Aurelien!
> 
> Aha, "vader btl", that is new to me!
> I tought Vader was that man dressed in black in Star Wars,
> Obi-Wan Kenobi's nemesis.
> That was a while ago, my kids were children,
> and Alec Guiness younger than Harrison Ford is today.
> Oh, how nostalgic code developers can get when it comes
> to naming things ...
> 
> If I am using "vader", it is totally inadvertent.
> There was no such a thing in Open MPI 1.6 and earlier.
> 
> Now that you mentioned, I can see lots of it in the 1.8.3
> ompi_info output.
> In addition, my stderr files show messages like this:
> 
> imb.e38352:[1,5]:[node13:16334] mca: bml: Not using sm btl to
> [[59987,1],26] on node node13 because vader btl has higher exclusivity
> (65536 > 65535)
> 
> So, you are right, "vader" is taking over and knocking off "sm" (and openib
> and everybody else).
> Darn Vader!
> Probably knem is going down the tubes along with sm, right?

Depends. If there is a reason to continue supporting knem then vader
will be updated to support it. I don't currently see a reason to at this
time though (since sm continues to live for now).

> I was used to sm, openib, self and tcp BTLs.
> I normally just do "btl = ^tcp" in the MCA parameters file,
> to stick to sm, openib, and self.
> 
> That worked fine in 1.6.5 (and earlier), and knem worked
> flawlessly there.
> The same settings in 1.8.3 don't bring up the knem functionality.
> So, this seems to be yet another change in 1.8.3 that I need to learn.
> 
> Can you or some other list subscriber elaborate a bit about
> this 'vader' btl?
> The Open MPI FAQ doesn't have anthing about it.
> What is it after all?
> Does it play the same role as "sm", i.e., an intra-node btl?
> Considering the name, is "vader" good or bad?
> Or better: In which circumstances is "vader" good and when is it bad?

Vader is a btl I originally wrote to support Cray's XPMEM shared memory
interface. It was designed to be cleaner than btl/sm have better small
message latency, bandwidth, and message rates. Because its latency is so
much better than sm I removed the XPMEM requirement and added CMA
support.

> Should I give in to the dark side of the force and keep "vader"
> turned on, or should I just do something like
> "btl = ^tcp,^vader" ?

You can turn off vader if you want to use knem. I would run some tests
to see if there is much of a difference between sm/knem and vader
though. I don't have any systems that have knem installed so I haven't
been able to run these tests myself. I would primarily focus on the
memory usage and the bandwidth.

-Nathan


pgpk1r7Q9MgIv.pgp
Description: PGP signature

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Nathan Hjelm 
And it doesn't support knem at this time. Probably never will because of
the existence of CMA.

-Nathan

On Thu, Oct 16, 2014 at 01:49:09PM -0700, Ralph Castain wrote:
> FWIW: vader is the default in 1.8
> 
> On Oct 16, 2014, at 1:40 PM, Aurélien Bouteiller  wrote:
> 
> > Are you sure you are not using the vader BTL ? 
> > 
> > Setting mca_btl_base_verbose and/or sm_verbose should spit out some knem 
> > initialization info. 
> > 
> > The CMA linux system (that ships with most 3.1x linux kernels) has similar 
> > features, and is also supported in sm.
> > 
> > Aurelien
> > --
> >  ~~~ Aurélien Bouteiller, Ph.D. ~~~
> > ~ Research Scientist @ ICL ~
> > The University of Tennessee, Innovative Computing Laboratory
> > 1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
> > tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
> > https://icl.cs.utk.edu/~bouteill/
> > 
> > 
> > 
> > 
> > Le 16 oct. 2014 à 16:35, Gus Correa  a écrit :
> > 
> >> Dear Open MPI developers
> >> 
> >> Well, I just can't keep my promises for too long ...
> >> So, here I am pestering you again, although this time
> >> it is not a request for more documentation.
> >> Hopefully it is something more legit.
> >> 
> >> I am having trouble using knem with Open MPI 1.8.3,
> >> and need your help.
> >> 
> >> I configured Open MPI 1.8.3 with knem.
> >> I had done the same with some builds of Open MPI 1.6.5 before.
> >> 
> >> When I build and launch the Intel MPI benchmarks (IMB)
> >> with Open MPI 1.6.5,
> >> 'cat /dev/knem'
> >> starts showing non-zero-and-growing statistics right away.
> >> 
> >> However, when I build and launch IMB with Open MPI 1.8.3,
> >> /dev/knem shows only zeros,
> >> no statistics growing, nothing.
> >> Knem just seems to be completely asleep.
> >> 
> >> So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
> >> at least not for me.
> >> 
> >> ***
> >> 
> >> The runtime environment related to knem is setup the
> >> same way on both OPMI releases.
> >> I tried setting it up both on the command line:
> >> 
> >> -mca btl_sm_eager_limit 32768 -mca btl_sm_knem_dma_min 1048576
> >> 
> >> and on the MCA parameter file:
> >> 
> >> btl_sm_use_knem = 1
> >> btl_sm_eager_limit = 32768
> >> btl_sm_knem_dma_min = 1048576
> >> 
> >> and the behavior is the same (i.e., knem is active in 1.6.5,
> >> but doesn't seem to be used by 1.8.3, as indicated by the
> >> /dev/knem statistics.)
> >> 
> >> ***
> >> 
> >> When I 'grep -i knem config.log', both 1.6.5 and 1.8.3 builds show:
> >> 
> >> #define OMPI_BTL_SM_HAVE_KNEM 1
> >> 
> >> suggesting that both configurations picked up knem correctly.
> >> 
> >> On the other hand, when I do 'ompi_info --all --all |grep knem',
> >> OMPI 1.6.5 shows "btl_sm_have_knem_support":
> >> 
> >> 'MCA btl: information "btl_sm_have_knem_support" (value: <1>, data source: 
> >> default value)  Whether this component supports the knem Linux kernel 
> >> module or not'
> >> 
> >> By contrast, in OMPI 1.8.3 ompi_info doesn't show this particular item 
> >> ("btl_sm_have_knem_support"),
> >> although the *other* 'btl sm knem' items are there,
> >> namely "btl_sm_use_knem","btl_sm_knem_dma_min", 
> >> "btl_sm_knem_max_simultaneous".
> >> 
> >> I am scratching my head to understand why a parameter with such a
> >> suggestive name ("btl_sm_have_knem_support"),
> >> so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
> >> somehow vanished from ompi_info in OMPI 1.8.3.
> >> 
> >> ***
> >> 
> >> Questions:
> >> 
> >> - Am I doing something totally wrong,
> >> perhaps with the knem runtime environment?
> >> 
> >> - Was knem somehow phased out in 1.8.3?
> >> 
> >> - Could there be a bad interaction with other runtime parameters that
> >> somehow is knocking out knem in 1.8.3?
> >> (FYI, besides knem, I'm just excluding the tcp btl, binding to core, and 
> >> reporting the bindings, which is exactly what I do on 1.6.5,
> >> although the runtime parameter syntax has changed.)
> >> 
> >> - Is knem inadvertently not being activated at runtime in OMPI 1.8.3?
> >> (i.e. a bug)
> >> 
> >> - Is there a way to increase verbosity to detect if knem is being
> >> used by OMPI?
> >> That would certainly help to check what is going on.
> >> I tried '-mca btl_base_verbose 30' but there was no trace of knem
> >> in sderr/stdout of either 1.6.5 or 1.8.3.
> >> So, the evidence I have that knem is
> >> active in 1.6.5 but not in 1.8.3 comes only from the statistics in
> >> /dev/knem.
> >> 
> >> ***
> >> 
> >> 
> >> Thank you,
> >> Gus Correa
> >> 
> >> ***
> >> 
> >> PS - As an aside, I also have some questions on the knem setup,
> >> which I mostly copied from the knem web site
> >> (hopefully Brice Goglin is listening ...):
> >> 
> >> - Is 32768 in 'btl_sm_eager_limit 32768' a good number,
> >> or should it be larger/smaller/something else?
> >> [OK, I know I should benchmark it, but exploring the whole parameter
> >> space takes long, so why not asking? ]

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Gus Correa 

Thank you, Aurelien!

Aha, "vader btl", that is new to me!
I tought Vader was that man dressed in black in Star Wars,
Obi-Wan Kenobi's nemesis.
That was a while ago, my kids were children,
and Alec Guiness younger than Harrison Ford is today.
Oh, how nostalgic code developers can get when it comes
to naming things ...

If I am using "vader", it is totally inadvertent.
There was no such a thing in Open MPI 1.6 and earlier.

Now that you mentioned, I can see lots of it in the 1.8.3
ompi_info output.
In addition, my stderr files show messages like this:

imb.e38352:[1,5]:[node13:16334] mca: bml: Not using sm btl to 
[[59987,1],26] on node node13 because vader btl has higher exclusivity 
(65536 > 65535)


So, you are right, "vader" is taking over and knocking off "sm" (and 
openib and everybody else).

Darn Vader!
Probably knem is going down the tubes along with sm, right?

I was used to sm, openib, self and tcp BTLs.
I normally just do "btl = ^tcp" in the MCA parameters file,
to stick to sm, openib, and self.

That worked fine in 1.6.5 (and earlier), and knem worked
flawlessly there.
The same settings in 1.8.3 don't bring up the knem functionality.
So, this seems to be yet another change in 1.8.3 that I need to learn.

Can you or some other list subscriber elaborate a bit about
this 'vader' btl?
The Open MPI FAQ doesn't have anthing about it.
What is it after all?
Does it play the same role as "sm", i.e., an intra-node btl?
Considering the name, is "vader" good or bad?
Or better: In which circumstances is "vader" good and when is it bad?
Should I give in to the dark side of the force and keep "vader"
turned on, or should I just do something like
"btl = ^tcp,^vader" ?

I am in CentOS 6.5, stock kernel 2.6.32, no 3.1,no  CMA linux,
so I believe I need knem for now.

I tried '-mca btl_base_verbose 30' but no knem information came out.

Many thanks,
Gus Correa

On 10/16/2014 04:40 PM, Aurélien Bouteiller wrote:

Are you sure you are not using the vader BTL ?

Setting mca_btl_base_verbose and/or sm_verbose should spit

out some knem initialization info.


The CMA linux system (that ships with most 3.1x linux kernels)

has similar features, and is also supported in sm.


Aurelien
--
   ~~~ Aurélien Bouteiller, Ph.D. ~~~
  ~ Research Scientist @ ICL ~
The University of Tennessee, Innovative Computing Laboratory
1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
https://icl.cs.utk.edu/~bouteill/




Le 16 oct. 2014 à 16:35, Gus Correa  a écrit :


Dear Open MPI developers

Well, I just can't keep my promises for too long ...
So, here I am pestering you again, although this time
it is not a request for more documentation.
Hopefully it is something more legit.

I am having trouble using knem with Open MPI 1.8.3,
and need your help.

I configured Open MPI 1.8.3 with knem.
I had done the same with some builds of Open MPI 1.6.5 before.

When I build and launch the Intel MPI benchmarks (IMB)
with Open MPI 1.6.5,
'cat /dev/knem'
starts showing non-zero-and-growing statistics right away.

However, when I build and launch IMB with Open MPI 1.8.3,
/dev/knem shows only zeros,
no statistics growing, nothing.
Knem just seems to be completely asleep.

So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
at least not for me.

***

The runtime environment related to knem is setup the
same way on both OPMI releases.
I tried setting it up both on the command line:

-mca btl_sm_eager_limit 32768 -mca btl_sm_knem_dma_min 1048576

and on the MCA parameter file:

btl_sm_use_knem = 1
btl_sm_eager_limit = 32768
btl_sm_knem_dma_min = 1048576

and the behavior is the same (i.e., knem is active in 1.6.5,
but doesn't seem to be used by 1.8.3, as indicated by the
/dev/knem statistics.)

***

When I 'grep -i knem config.log', both 1.6.5 and 1.8.3 builds show:

#define OMPI_BTL_SM_HAVE_KNEM 1

suggesting that both configurations picked up knem correctly.

On the other hand, when I do 'ompi_info --all --all |grep knem',
OMPI 1.6.5 shows "btl_sm_have_knem_support":

'MCA btl: information "btl_sm_have_knem_support" (value: <1>, data source: 
default value)  Whether this component supports the knem Linux kernel module or not'

By contrast, in OMPI 1.8.3 ompi_info doesn't show this particular item 
("btl_sm_have_knem_support"),
although the *other* 'btl sm knem' items are there,
namely "btl_sm_use_knem","btl_sm_knem_dma_min", "btl_sm_knem_max_simultaneous".

I am scratching my head to understand why a parameter with such a
suggestive name ("btl_sm_have_knem_support"),
so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
somehow vanished from ompi_info in OMPI 1.8.3.

***

Questions:

- Am I doing something totally wrong,
perhaps with the knem runtime environment?

- Was knem somehow phased out in 1.8.3?

- Could there be a bad interaction with other runtime parameters that
somehow is knocking out knem in 1.8.3?
(FYI, besides knem, I'm just

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Ralph Castain 
FWIW: vader is the default in 1.8

On Oct 16, 2014, at 1:40 PM, Aurélien Bouteiller  wrote:

> Are you sure you are not using the vader BTL ? 
> 
> Setting mca_btl_base_verbose and/or sm_verbose should spit out some knem 
> initialization info. 
> 
> The CMA linux system (that ships with most 3.1x linux kernels) has similar 
> features, and is also supported in sm.
> 
> Aurelien
> --
>  ~~~ Aurélien Bouteiller, Ph.D. ~~~
> ~ Research Scientist @ ICL ~
> The University of Tennessee, Innovative Computing Laboratory
> 1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
> tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
> https://icl.cs.utk.edu/~bouteill/
> 
> 
> 
> 
> Le 16 oct. 2014 à 16:35, Gus Correa  a écrit :
> 
>> Dear Open MPI developers
>> 
>> Well, I just can't keep my promises for too long ...
>> So, here I am pestering you again, although this time
>> it is not a request for more documentation.
>> Hopefully it is something more legit.
>> 
>> I am having trouble using knem with Open MPI 1.8.3,
>> and need your help.
>> 
>> I configured Open MPI 1.8.3 with knem.
>> I had done the same with some builds of Open MPI 1.6.5 before.
>> 
>> When I build and launch the Intel MPI benchmarks (IMB)
>> with Open MPI 1.6.5,
>> 'cat /dev/knem'
>> starts showing non-zero-and-growing statistics right away.
>> 
>> However, when I build and launch IMB with Open MPI 1.8.3,
>> /dev/knem shows only zeros,
>> no statistics growing, nothing.
>> Knem just seems to be completely asleep.
>> 
>> So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
>> at least not for me.
>> 
>> ***
>> 
>> The runtime environment related to knem is setup the
>> same way on both OPMI releases.
>> I tried setting it up both on the command line:
>> 
>> -mca btl_sm_eager_limit 32768 -mca btl_sm_knem_dma_min 1048576
>> 
>> and on the MCA parameter file:
>> 
>> btl_sm_use_knem = 1
>> btl_sm_eager_limit = 32768
>> btl_sm_knem_dma_min = 1048576
>> 
>> and the behavior is the same (i.e., knem is active in 1.6.5,
>> but doesn't seem to be used by 1.8.3, as indicated by the
>> /dev/knem statistics.)
>> 
>> ***
>> 
>> When I 'grep -i knem config.log', both 1.6.5 and 1.8.3 builds show:
>> 
>> #define OMPI_BTL_SM_HAVE_KNEM 1
>> 
>> suggesting that both configurations picked up knem correctly.
>> 
>> On the other hand, when I do 'ompi_info --all --all |grep knem',
>> OMPI 1.6.5 shows "btl_sm_have_knem_support":
>> 
>> 'MCA btl: information "btl_sm_have_knem_support" (value: <1>, data source: 
>> default value)  Whether this component supports the knem Linux kernel module 
>> or not'
>> 
>> By contrast, in OMPI 1.8.3 ompi_info doesn't show this particular item 
>> ("btl_sm_have_knem_support"),
>> although the *other* 'btl sm knem' items are there,
>> namely "btl_sm_use_knem","btl_sm_knem_dma_min", 
>> "btl_sm_knem_max_simultaneous".
>> 
>> I am scratching my head to understand why a parameter with such a
>> suggestive name ("btl_sm_have_knem_support"),
>> so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
>> somehow vanished from ompi_info in OMPI 1.8.3.
>> 
>> ***
>> 
>> Questions:
>> 
>> - Am I doing something totally wrong,
>> perhaps with the knem runtime environment?
>> 
>> - Was knem somehow phased out in 1.8.3?
>> 
>> - Could there be a bad interaction with other runtime parameters that
>> somehow is knocking out knem in 1.8.3?
>> (FYI, besides knem, I'm just excluding the tcp btl, binding to core, and 
>> reporting the bindings, which is exactly what I do on 1.6.5,
>> although the runtime parameter syntax has changed.)
>> 
>> - Is knem inadvertently not being activated at runtime in OMPI 1.8.3?
>> (i.e. a bug)
>> 
>> - Is there a way to increase verbosity to detect if knem is being
>> used by OMPI?
>> That would certainly help to check what is going on.
>> I tried '-mca btl_base_verbose 30' but there was no trace of knem
>> in sderr/stdout of either 1.6.5 or 1.8.3.
>> So, the evidence I have that knem is
>> active in 1.6.5 but not in 1.8.3 comes only from the statistics in
>> /dev/knem.
>> 
>> ***
>> 
>> 
>> Thank you,
>> Gus Correa
>> 
>> ***
>> 
>> PS - As an aside, I also have some questions on the knem setup,
>> which I mostly copied from the knem web site
>> (hopefully Brice Goglin is listening ...):
>> 
>> - Is 32768 in 'btl_sm_eager_limit 32768' a good number,
>> or should it be larger/smaller/something else?
>> [OK, I know I should benchmark it, but exploring the whole parameter
>> space takes long, so why not asking? ]
>> 
>> - Is it worth using 'btl_sm_knem_dma_min 1048576'?
>> [I think I read somewhere that this dma engine offload
>> is an Intel thing, not AMD.]
>> 
>> - How about btl_sm_knem_max_simultaneous?
>> That one is not mentioned in the knem web site.
>> Should I leave it default to zero or set it to 1? 2? 4? Something else?
>> 
>> 
>> Thanks again,
>> Gus Correa
>> ___
>> users mailing list
>>

Re: [OMPI users] knem in Open MPI 1.8.3

2014-10-16 Thread Aurélien Bouteiller 
Are you sure you are not using the vader BTL ? 

Setting mca_btl_base_verbose and/or sm_verbose should spit out some knem 
initialization info. 

The CMA linux system (that ships with most 3.1x linux kernels) has similar 
features, and is also supported in sm.

Aurelien
--
  ~~~ Aurélien Bouteiller, Ph.D. ~~~
 ~ Research Scientist @ ICL ~
The University of Tennessee, Innovative Computing Laboratory
1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
https://icl.cs.utk.edu/~bouteill/




Le 16 oct. 2014 à 16:35, Gus Correa  a écrit :

> Dear Open MPI developers
> 
> Well, I just can't keep my promises for too long ...
> So, here I am pestering you again, although this time
> it is not a request for more documentation.
> Hopefully it is something more legit.
> 
> I am having trouble using knem with Open MPI 1.8.3,
> and need your help.
> 
> I configured Open MPI 1.8.3 with knem.
> I had done the same with some builds of Open MPI 1.6.5 before.
> 
> When I build and launch the Intel MPI benchmarks (IMB)
> with Open MPI 1.6.5,
> 'cat /dev/knem'
> starts showing non-zero-and-growing statistics right away.
> 
> However, when I build and launch IMB with Open MPI 1.8.3,
> /dev/knem shows only zeros,
> no statistics growing, nothing.
> Knem just seems to be completely asleep.
> 
> So, my conclusion is that somehow knem is not working with OMPI 1.8.3,
> at least not for me.
> 
> ***
> 
> The runtime environment related to knem is setup the
> same way on both OPMI releases.
> I tried setting it up both on the command line:
> 
> -mca btl_sm_eager_limit 32768 -mca btl_sm_knem_dma_min 1048576
> 
> and on the MCA parameter file:
> 
> btl_sm_use_knem = 1
> btl_sm_eager_limit = 32768
> btl_sm_knem_dma_min = 1048576
> 
> and the behavior is the same (i.e., knem is active in 1.6.5,
> but doesn't seem to be used by 1.8.3, as indicated by the
> /dev/knem statistics.)
> 
> ***
> 
> When I 'grep -i knem config.log', both 1.6.5 and 1.8.3 builds show:
> 
> #define OMPI_BTL_SM_HAVE_KNEM 1
> 
> suggesting that both configurations picked up knem correctly.
> 
> On the other hand, when I do 'ompi_info --all --all |grep knem',
> OMPI 1.6.5 shows "btl_sm_have_knem_support":
> 
> 'MCA btl: information "btl_sm_have_knem_support" (value: <1>, data source: 
> default value)  Whether this component supports the knem Linux kernel module 
> or not'
> 
> By contrast, in OMPI 1.8.3 ompi_info doesn't show this particular item 
> ("btl_sm_have_knem_support"),
> although the *other* 'btl sm knem' items are there,
> namely "btl_sm_use_knem","btl_sm_knem_dma_min", 
> "btl_sm_knem_max_simultaneous".
> 
> I am scratching my head to understand why a parameter with such a
> suggestive name ("btl_sm_have_knem_support"),
> so similar to the OMPI_BTL_SM_HAVE_KNEM cpp macro,
> somehow vanished from ompi_info in OMPI 1.8.3.
> 
> ***
> 
> Questions:
> 
> - Am I doing something totally wrong,
> perhaps with the knem runtime environment?
> 
> - Was knem somehow phased out in 1.8.3?
> 
> - Could there be a bad interaction with other runtime parameters that
> somehow is knocking out knem in 1.8.3?
> (FYI, besides knem, I'm just excluding the tcp btl, binding to core, and 
> reporting the bindings, which is exactly what I do on 1.6.5,
> although the runtime parameter syntax has changed.)
> 
> - Is knem inadvertently not being activated at runtime in OMPI 1.8.3?
> (i.e. a bug)
> 
> - Is there a way to increase verbosity to detect if knem is being
> used by OMPI?
> That would certainly help to check what is going on.
> I tried '-mca btl_base_verbose 30' but there was no trace of knem
> in sderr/stdout of either 1.6.5 or 1.8.3.
> So, the evidence I have that knem is
> active in 1.6.5 but not in 1.8.3 comes only from the statistics in
> /dev/knem.
> 
> ***
> 
> 
> Thank you,
> Gus Correa
> 
> ***
> 
> PS - As an aside, I also have some questions on the knem setup,
> which I mostly copied from the knem web site
> (hopefully Brice Goglin is listening ...):
> 
> - Is 32768 in 'btl_sm_eager_limit 32768' a good number,
> or should it be larger/smaller/something else?
> [OK, I know I should benchmark it, but exploring the whole parameter
> space takes long, so why not asking? ]
> 
> - Is it worth using 'btl_sm_knem_dma_min 1048576'?
> [I think I read somewhere that this dma engine offload
> is an Intel thing, not AMD.]
> 
> - How about btl_sm_knem_max_simultaneous?
> That one is not mentioned in the knem web site.
> Should I leave it default to zero or set it to 1? 2? 4? Something else?
> 
> 
> Thanks again,
> Gus Correa
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/10/25511.php



signature.asc
Description: Message signed with OpenPGP using GPGMail