Re: [OMPI users] What Red Hat Enterprise/CentOS NUMA libraries are recommended/required for OpenMPI?

[Dave Love]

"Jeff Squyres (jsquyres)"  writes:

> I think Dave's point is that numactl-devel (and numactl) is only needed for 
> *building* Open MPI.  Users only need numactl to *run* Open MPI.

Yes.  However, I guess the basic problem is that the component fails to
load for want of libhwloc, either because (the right soname of) it isn't
present or there's a problem with numactl or another of its
dependencies.  (That won't happen if you use a packaged version, of
course.)

There are three instances of "libnumactl and libnumactl-devel" in the
release .txt files which I think are wrong.  I don't know on what system
such package names exist (not Debian or Red Hat-ish) and numactl isn't
even relevant to all Linux-based systems (e.g. not s390 or arm in
Fedora).  I'd replace the message with one saying that a compatible
version of libhwloc and its dependencies needs to be available, assuming
I've got that right.  The -devel/-dev package is surely not required
anyway.

[In the context of a recent version of SGE, I don't know how the support
can be missing; the execd and shepherd should be built with hwloc, and
there's a test in the rc script that the shepherd will actually load.]

Re: [OMPI users] What Red Hat Enterprise/CentOS NUMA libraries are recommended/required for OpenMPI?

[Jeff Squyres (jsquyres)]

I think Dave's point is that numactl-devel (and numactl) is only needed for 
*building* Open MPI.  Users only need numactl to *run* Open MPI.

Specifically, numactl-devel contains the .h files we need to compile OMPI 
against libnumactl:

$ rpm -ql numactl-devel
/usr/include/numa.h
/usr/include/numacompat1.h
/usr/include/numaif.h
/usr/lib64/libnuma.a
/usr/lib64/libnuma.so
/usr/share/man/man3/numa.3.gz

Note that the .so is a sym link to .so.1, in the main numactl package:

$ rpm -ql numactl
/usr/bin/memhog
/usr/bin/migratepages
/usr/bin/migspeed
/usr/bin/numactl
/usr/bin/numademo
/usr/bin/numastat
/usr/lib64/libnuma.so.1
/usr/share/man/man8/migratepages.8.gz
/usr/share/man/man8/migspeed.8.gz
/usr/share/man/man8/numactl.8.gz
/usr/share/man/man8/numastat.8.gz


> On Aug 11, 2015, at 12:42 PM, Ralph Castain  wrote:
> 
> Because only the devel package includes the necessary pieces to set memory 
> affinity.
> 
> 
> On Tue, Aug 11, 2015 at 9:37 AM, Dave Love  wrote:
> Ralph Castain  writes:
> 
> > Hi Bill
> >
> > You need numactl-devel on the nodes. Not having them means we cannot ensure
> > memory is bound local to the procs, which will hurt performance but not
> > much else. There is an MCA param to turn off the warnings if you choose not
> > to install the libs: hwloc_base_mem_bind_failure_action=silent
> 
> Why should you need the -devel package on the compute nodes?  (It only
> contains the .h and .so files.)  The RHEL and Fedora packages don't
> require it and work.
> 
> [For an up-to-date OMPI, you can rebuild the package against the current
> tarball, at least after the chaos caused by RHEL 6.6 updating
> incompatibly to 1.8.  Otherwise use the Fedora packaging, which is kept
> quite current.]
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/08/27427.php
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/08/27428.php


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] What Red Hat Enterprise/CentOS NUMA libraries are recommended/required for OpenMPI?

[Ralph Castain]

Because only the devel package includes the necessary pieces to set memory
affinity.


On Tue, Aug 11, 2015 at 9:37 AM, Dave Love  wrote:

> Ralph Castain  writes:
>
> > Hi Bill
> >
> > You need numactl-devel on the nodes. Not having them means we cannot
> ensure
> > memory is bound local to the procs, which will hurt performance but not
> > much else. There is an MCA param to turn off the warnings if you choose
> not
> > to install the libs: hwloc_base_mem_bind_failure_action=silent
>
> Why should you need the -devel package on the compute nodes?  (It only
> contains the .h and .so files.)  The RHEL and Fedora packages don't
> require it and work.
>
> [For an up-to-date OMPI, you can rebuild the package against the current
> tarball, at least after the chaos caused by RHEL 6.6 updating
> incompatibly to 1.8.  Otherwise use the Fedora packaging, which is kept
> quite current.]
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/08/27427.php
>

Re: [OMPI users] What Red Hat Enterprise/CentOS NUMA libraries are recommended/required for OpenMPI?

[Dave Love]

Ralph Castain  writes:

> Hi Bill
>
> You need numactl-devel on the nodes. Not having them means we cannot ensure
> memory is bound local to the procs, which will hurt performance but not
> much else. There is an MCA param to turn off the warnings if you choose not
> to install the libs: hwloc_base_mem_bind_failure_action=silent

Why should you need the -devel package on the compute nodes?  (It only
contains the .h and .so files.)  The RHEL and Fedora packages don't
require it and work.

[For an up-to-date OMPI, you can rebuild the package against the current
tarball, at least after the chaos caused by RHEL 6.6 updating
incompatibly to 1.8.  Otherwise use the Fedora packaging, which is kept
quite current.]

Re: [OMPI users] What Red Hat Enterprise/CentOS NUMA libraries are recommended/required for OpenMPI?

[Ralph Castain]

Hi Bill

You need numactl-devel on the nodes. Not having them means we cannot ensure
memory is bound local to the procs, which will hurt performance but not
much else. There is an MCA param to turn off the warnings if you choose not
to install the libs: hwloc_base_mem_bind_failure_action=silent

Ralph


On Tue, Aug 4, 2015 at 10:36 PM, Lane, William 
wrote:

> I'm running OpenMPI 1.8.7 tests on a mixed bag cluster of various systems
> under CentOS 6.3, I've been intermittently getting warnings about not
> having
> the proper NUMA libraries installed. Which NUMA libraries should be
> installed
> for CentOS 6.3 and OpenMPI 1.8.7?
>
> Here's what I currently have installed:
>
> yum list installed *numa*
> numactl.x86_64
>
> Here's the list of available NUMA libraries for CentOS 6.3:
>
> listed via: yum search numa | less
> numactl-devel.i686 : Development package for building Applications
> that use numa
> numactl-devel.x86_64 : Development package for building Applications
> that use
>  : numa
> numad.x86_64 : NUMA user daemon
> numactl.i686 : Library for tuning for Non Uniform Memory Access
> machines
> numactl.x86_64 : Library for tuning for Non Uniform Memory Access
> machines
>
> Also, since this cluster actually has working NUMA nodes, could the lack
> of the proper NUMA libraries being installed cause any issues?
>
> -Bill L.
> IMPORTANT WARNING: This message is intended for the use of the person or
> entity to which it is addressed and may contain information that is
> privileged and confidential, the disclosure of which is governed by
> applicable law. If the reader of this message is not the intended
> recipient, or the employee or agent responsible for delivering it to the
> intended recipient, you are hereby notified that any dissemination,
> distribution or copying of this information is strictly prohibited. Thank
> you for your cooperation.
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/08/27394.php
>

[OMPI users] What Red Hat Enterprise/CentOS NUMA libraries are recommended/required for OpenMPI?

[Lane, William]

I'm running OpenMPI 1.8.7 tests on a mixed bag cluster of various systems
under CentOS 6.3, I've been intermittently getting warnings about not having
the proper NUMA libraries installed. Which NUMA libraries should be installed
for CentOS 6.3 and OpenMPI 1.8.7?

Here's what I currently have installed:

yum list installed *numa*
numactl.x86_64

Here's the list of available NUMA libraries for CentOS 6.3:

listed via: yum search numa | less
numactl-devel.i686 : Development package for building Applications that use 
numa
numactl-devel.x86_64 : Development package for building Applications that 
use
 : numa
numad.x86_64 : NUMA user daemon
numactl.i686 : Library for tuning for Non Uniform Memory Access machines
numactl.x86_64 : Library for tuning for Non Uniform Memory Access machines

Also, since this cluster actually has working NUMA nodes, could the lack
of the proper NUMA libraries being installed cause any issues?

-Bill L.
IMPORTANT WARNING: This message is intended for the use of the person or entity 
to which it is addressed and may contain information that is privileged and 
confidential, the disclosure of which is governed by applicable law. If the 
reader of this message is not the intended recipient, or the employee or agent 
responsible for delivering it to the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this information is 
strictly prohibited. Thank you for your cooperation.