Re: [OMPI users] Suggestions on multi-compiler/multi-mpi build?

[Katherine Holcomb]

On Fri, 2007-11-30 at 19:05 -0800, Jim Kusznir wrote:
> Thank you for your response!
> 
> Just to clarify some things for my understanding:
> 
> Do users load a single module that specifies both compiler and mpi
> version (as opposed to loading two different modules, one for
> complier, and one for mpi)?

We combine compiler and MPI because it's necessary that the
mpifcc/cxx/f90 be compatible with the compiler, so the correct compiler
needs to be in the user's path, and it's just easier on the users to
have them load a single module.  A module can load another module, so if
you create a module for, say, the PGI compilers, the appropriate OpenMPI
module can load that module itself.

Here's an example, my ompi-pgi script (note: all my scripts were cribbed
from old ROCKS scripts; ROCKS doesn't include modules by default anymore
but they used to provide it):


#%
Module3.0#
##
## CRAYfw mpich
##
proc ModulesHelp { } {
global version MPIhome

puts stderr "\tCRAY freeware: OpenMPI\n"
puts stderr "\tThis module loads the lastest versions of OpenMPI.\n"
puts stderr "\n\tThis adds $OpenMPI/* to several of the"
puts stderr "\tenvironment variables."
puts stderr "\n\tVersion $version\n"
}

module-whatis   "loads the OpenMPI environment for PGI over Ethernet"

# for Tcl script use only
set version 1.2
set MPIhome /opt/openmpi-1.2/ethernet/pgi

prepend-pathPATH$MPIhome/bin
prepend-pathINCLUDE $MPIhome/include
prepend-pathINFOPATH$MPIhome/share
prepend-pathLD_LIBRARY_PATH $MPIhome/lib

setenv OMPI_MPIBIN $MPIhome
setenv OMPI_MPIHOME $MPIhome
setenv OMPI_MPIF77 "pgf77"
setenv OMPI_MPIF90 "pgf90"
setenv OMPI_MPICC  "pgcc"
setenv OMPI_MPICXX "pgCC"

if [ module-info mode load ] {

if  {! [ is-loaded pgi ] } { module load pgi }

if {  [ is-loaded ompi-pgi ] } { 
puts stderr "\tModule ompi-pgi already loaded\n" 
}
if {  [ is-loaded ompi-gnu ] } { 
puts stderr "\tModule ompi-gnu superceded\n" 
}
if {  [ is-loaded ompi-intel ] } { 
puts stderr "\tModule ompi-intel superceded\n" 
}
if {  [ is-loaded mpich-eth-pgi   ] } {
puts stderr "\tModule mpich-eth-pgi environment superceded.\n"
}

if {  [ is-loaded mpich-eth-gnu   ] } {
puts stderr "\tModule mpich-eth-gnu environment superceded.\n"
}
if {  [ is-loaded mpich-eth-intel   ] } {
puts stderr "\tModule mpich-eth-intel environment superceded.\n"
}
}

> 
> So, if the MPI implementations need to be built separately for each
> compiler, what about other software packages that users' link against?
>  For example, as far as I know, there are not seperate versions of
> libc for each compiler (but I may be ignorant hereplease correct
> me if I'm wrong).  How about other packages that code #includes?  How
> do I know when something needs to be compiled separately vs one
> version for everyone?

We build a distinct version at least for every library that has F90/95
bindings.  Probably a good idea for C++ as well.

> 
> Another thing that I'm still not quite clear on:  Can I have all
> versions of the various MPI libraries in the ld path at once (so the
> user doesn't have to "select" the correct one somehow prior to code
> execution)?

I'll let somebody who is more of a real system administrator answer
that.

> 
> Are there any tricks to building/installing all this stuff in a fairly
> automated manner?
> 

We use ROCKS.  You have to create an rpm for everything, but once you do
then propagating it is automated.

-- 
Katherine Holcomb, Ph.D.kholc...@virginia.edu
Research Computing Support Group - ITC  Office Phone: (434) 982-5948
I046 BSEL, Clark HallCenter Phone: (434) 243-8799
University of Virginia 22904

Re: [OMPI users] Suggestions on multi-compiler/multi-mpi build?

[Jim Kusznir]

Thank you for your response!

Just to clarify some things for my understanding:

Do users load a single module that specifies both compiler and mpi
version (as opposed to loading two different modules, one for
complier, and one for mpi)?

So, if the MPI implementations need to be built separately for each
compiler, what about other software packages that users' link against?
 For example, as far as I know, there are not seperate versions of
libc for each compiler (but I may be ignorant hereplease correct
me if I'm wrong).  How about other packages that code #includes?  How
do I know when something needs to be compiled separately vs one
version for everyone?

Another thing that I'm still not quite clear on:  Can I have all
versions of the various MPI libraries in the ld path at once (so the
user doesn't have to "select" the correct one somehow prior to code
execution)?

Are there any tricks to building/installing all this stuff in a fairly
automated manner?

Thanks again!
--Jim

On Nov 15, 2007 9:13 AM, Katherine Holcomb <kholc...@virginia.edu> wrote:
> We have almost exactly the situation you describe with our clusters.  I'm
> not the system administrator but I am the one who actually writes the module
> scripts.
>
> It is best to compile OpenMPI (and any other MPI) with each compiler
> separately; this is especially necessary for the Fortran and C++ bindings.
> We simply have a directory layout like
>
> /opt/openmpi/intel
> /opt/openmpi/pgi
> /opt/openmpi/gnu
>
> etc.
>
> To compile OpenMPI with a different compiler, you can use environment
> variables such as (for the Intel compiler)
> FC=ifort
> F90=ifort
> CC=icc
> CXX=icpc
> You set these within the %build part of the rpm spec file.
>
> Similarly for MPICH2.  (I would recommend MPICH2 over MPICH1 at this point.)
>
> Users load a module to select a compiler+MPI combination.  Loading the
> appropriate module sets the path for the version of mpicc/cxx/f90 chosen by
> the user.  That in turn knows where its associated libraries live.
>
> Most users link MPI libraries statically.  If dynamic libraries are needed
> then they can be added to LD_LIBRARY_PATH by the module script.
>
> I am away from my office until after the Thanksgiving holiday but if you
> email me personally
>  then, I can send you some sample module scripts.  I can also send you some
> sample spec files for the rpms we use.
>
>
> - Original Message -----
> From: "Jim Kusznir" <jkusz...@gmail.com>
> To: <us...@open-mpi.org>
> Sent: Thursday, November 15, 2007 11:54 AM
> Subject: [OMPI users] Suggestions on multi-compiler/multi-mpi build?
>
>
> > Hi all:
> >
> > I'm trying to set up a cluster for a group of users with very
> > different needs.  So far, it looks like I need gcc, pgi, and intel to
> > work with openmpi and mpich, with each user able to control what
> > combination they get.  This is turning out to be much more difficult
> > than I expected.
> >
> > Someone has pointed me to enviornment-modules ("Modules"), which looks
> > like it will be a critical part of the solution.  I even noticed that
> > the provided openmpi.spec file has some direct support for modules,
> > which makes me quite happy!
> >
> > However, I still have many questions about how to set things up.
> >
> > First, I get the impression that openmpi will need to be compiled with
> > each compiler that will use it.  If this is true, I'm not quite sure
> > how to go about it.  I could install in different directories for the
> > user commands, but what about the libraries?  I don't think I have a
> > feesable way of selecting which library to use on the fly on the
> > entire cluster for each user, so it seems like it would be better to
> > have all the libraries available.  In addition, I will need RPMs to
> > deploy efficiently on the cluster.  I suspect I can change the
> > versioning info and build with each compiler, but at this point, I
> > don't even know how to reliably select what compiler rpmbuild will use
> > (I've only succeeded in using gcc).
> >
> > Finally, using modules, how do I set it up so that if a user changes
> > compilers, but stays with openmpi, it will load the correct openmpi
> > paths?  I know I can set up the openmpi module file to load after the
> > compiler module and based on that select different paths based on the
> > currently-loaded compiler module.  If the user changes the compiler
> > module, will that cause the mpi module to also be reloaded so the new
> > settings will be loaded?  Or do I need this at all?
> >
> > Thanks for all your help!
> > --Jim
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] Suggestions on multi-compiler/multi-mpi build?

[Jim Kusznir]

So another question on this topic:

I'm running on a 64-bit cluster.  Is it important/needed/useful to
maintain 32-bit and 64-bit versions of openmpi and such?  At the
moment, I'm using Rocks' default openmpi, which includes both 32 and
64-bit versions of the libraries.  Yet, so far, my attempt to build a
32-bit on the 64-bit system has proven very difficult, and I finally
resolved to just install a 32-bit build host to build the 32-bit
versions if required.

Thanks!

Oh, and I'd be very grateful for any samples and automation scripts to
help with the whole setup.  I'm pretty new to "real cluster
management" :)

Thanks again!

--Jim

On 11/15/07, Brock Palen  wrote:
> Really modules is the only way to go, we use it to maintain no less
> than 12 versions of openmpi compiled with pgi, gcc, nag, and intel.
>
> Yes as the other reply said just set LD_LIBRARY_PATH  you can use
>
> ldd executable
>
> to see which libraries the executable is going to use under the
> current environment.  We also have uses submit to torque with
> #PBS -V
>
> which copies the current environment with the job, this allows users
> to submit and have multiple jobs running each with their own
> environment.  If your using PBS or torque look at the qsub man page.
>
> Another addition we came up with was after we load a default set a
> modules (pbs, a compiler , mpi library)  we look for a file in
>
> $HOME/privatemodules/default
>
> which just has a list of module commands.  This allows users to
> change their default modules for every login.
> For example i have in my default file
>
> module load git
>
> Because i use git for managing files.
>
> We have done other hacks, our webpage of software is built out of
> cron from the module files:
> http://cac.engin.umich.edu/resources/systems/nyxV2/software.html
>
> So we dont maintain software lists online, we just generate it
> dynamically.
> Modules is the admins best friend.
>
> Brock Palen
> Center for Advanced Computing
> bro...@umich.edu
> (734)936-1985
>
>
> On Nov 15, 2007, at 8:54 AM, Jim Kusznir wrote:
>
> > Hi all:
> >
> > I'm trying to set up a cluster for a group of users with very
> > different needs.  So far, it looks like I need gcc, pgi, and intel to
> > work with openmpi and mpich, with each user able to control what
> > combination they get.  This is turning out to be much more difficult
> > than I expected.
> >
> > Someone has pointed me to enviornment-modules ("Modules"), which looks
> > like it will be a critical part of the solution.  I even noticed that
> > the provided openmpi.spec file has some direct support for modules,
> > which makes me quite happy!
> >
> > However, I still have many questions about how to set things up.
> >
> > First, I get the impression that openmpi will need to be compiled with
> > each compiler that will use it.  If this is true, I'm not quite sure
> > how to go about it.  I could install in different directories for the
> > user commands, but what about the libraries?  I don't think I have a
> > feesable way of selecting which library to use on the fly on the
> > entire cluster for each user, so it seems like it would be better to
> > have all the libraries available.  In addition, I will need RPMs to
> > deploy efficiently on the cluster.  I suspect I can change the
> > versioning info and build with each compiler, but at this point, I
> > don't even know how to reliably select what compiler rpmbuild will use
> > (I've only succeeded in using gcc).
> >
> > Finally, using modules, how do I set it up so that if a user changes
> > compilers, but stays with openmpi, it will load the correct openmpi
> > paths?  I know I can set up the openmpi module file to load after the
> > compiler module and based on that select different paths based on the
> > currently-loaded compiler module.  If the user changes the compiler
> > module, will that cause the mpi module to also be reloaded so the new
> > settings will be loaded?  Or do I need this at all?
> >
> > Thanks for all your help!
> > --Jim
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
> >
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] Suggestions on multi-compiler/multi-mpi build?

[Brock Palen]

Really modules is the only way to go, we use it to maintain no less  
than 12 versions of openmpi compiled with pgi, gcc, nag, and intel.


Yes as the other reply said just set LD_LIBRARY_PATH  you can use

ldd executable

to see which libraries the executable is going to use under the  
current environment.  We also have uses submit to torque with

#PBS -V

which copies the current environment with the job, this allows users  
to submit and have multiple jobs running each with their own  
environment.  If your using PBS or torque look at the qsub man page.


Another addition we came up with was after we load a default set a  
modules (pbs, a compiler , mpi library)  we look for a file in


$HOME/privatemodules/default

which just has a list of module commands.  This allows users to  
change their default modules for every login.

For example i have in my default file

module load git

Because i use git for managing files.

We have done other hacks, our webpage of software is built out of  
cron from the module files:

http://cac.engin.umich.edu/resources/systems/nyxV2/software.html

So we dont maintain software lists online, we just generate it  
dynamically.

Modules is the admins best friend.

Brock Palen
Center for Advanced Computing
bro...@umich.edu
(734)936-1985


On Nov 15, 2007, at 8:54 AM, Jim Kusznir wrote:


Hi all:

I'm trying to set up a cluster for a group of users with very
different needs.  So far, it looks like I need gcc, pgi, and intel to
work with openmpi and mpich, with each user able to control what
combination they get.  This is turning out to be much more difficult
than I expected.

Someone has pointed me to enviornment-modules ("Modules"), which looks
like it will be a critical part of the solution.  I even noticed that
the provided openmpi.spec file has some direct support for modules,
which makes me quite happy!

However, I still have many questions about how to set things up.

First, I get the impression that openmpi will need to be compiled with
each compiler that will use it.  If this is true, I'm not quite sure
how to go about it.  I could install in different directories for the
user commands, but what about the libraries?  I don't think I have a
feesable way of selecting which library to use on the fly on the
entire cluster for each user, so it seems like it would be better to
have all the libraries available.  In addition, I will need RPMs to
deploy efficiently on the cluster.  I suspect I can change the
versioning info and build with each compiler, but at this point, I
don't even know how to reliably select what compiler rpmbuild will use
(I've only succeeded in using gcc).

Finally, using modules, how do I set it up so that if a user changes
compilers, but stays with openmpi, it will load the correct openmpi
paths?  I know I can set up the openmpi module file to load after the
compiler module and based on that select different paths based on the
currently-loaded compiler module.  If the user changes the compiler
module, will that cause the mpi module to also be reloaded so the new
settings will be loaded?  Or do I need this at all?

Thanks for all your help!
--Jim
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Suggestions on multi-compiler/multi-mpi build?

[Katherine Holcomb]

We have almost exactly the situation you describe with our clusters.  I'm 
not the system administrator but I am the one who actually writes the module 
scripts.


It is best to compile OpenMPI (and any other MPI) with each compiler 
separately; this is especially necessary for the Fortran and C++ bindings. 
We simply have a directory layout like


/opt/openmpi/intel
/opt/openmpi/pgi
/opt/openmpi/gnu

etc.

To compile OpenMPI with a different compiler, you can use environment 
variables such as (for the Intel compiler)

FC=ifort
F90=ifort
CC=icc
CXX=icpc
You set these within the %build part of the rpm spec file.

Similarly for MPICH2.  (I would recommend MPICH2 over MPICH1 at this point.)

Users load a module to select a compiler+MPI combination.  Loading the 
appropriate module sets the path for the version of mpicc/cxx/f90 chosen by 
the user.  That in turn knows where its associated libraries live.


Most users link MPI libraries statically.  If dynamic libraries are needed 
then they can be added to LD_LIBRARY_PATH by the module script.


I am away from my office until after the Thanksgiving holiday but if you 
email me personally
then, I can send you some sample module scripts.  I can also send you some 
sample spec files for the rpms we use.


- Original Message - 
From: "Jim Kusznir" <jkusz...@gmail.com>

To: <us...@open-mpi.org>
Sent: Thursday, November 15, 2007 11:54 AM
Subject: [OMPI users] Suggestions on multi-compiler/multi-mpi build?



Hi all:

I'm trying to set up a cluster for a group of users with very
different needs.  So far, it looks like I need gcc, pgi, and intel to
work with openmpi and mpich, with each user able to control what
combination they get.  This is turning out to be much more difficult
than I expected.

Someone has pointed me to enviornment-modules ("Modules"), which looks
like it will be a critical part of the solution.  I even noticed that
the provided openmpi.spec file has some direct support for modules,
which makes me quite happy!

However, I still have many questions about how to set things up.

First, I get the impression that openmpi will need to be compiled with
each compiler that will use it.  If this is true, I'm not quite sure
how to go about it.  I could install in different directories for the
user commands, but what about the libraries?  I don't think I have a
feesable way of selecting which library to use on the fly on the
entire cluster for each user, so it seems like it would be better to
have all the libraries available.  In addition, I will need RPMs to
deploy efficiently on the cluster.  I suspect I can change the
versioning info and build with each compiler, but at this point, I
don't even know how to reliably select what compiler rpmbuild will use
(I've only succeeded in using gcc).

Finally, using modules, how do I set it up so that if a user changes
compilers, but stays with openmpi, it will load the correct openmpi
paths?  I know I can set up the openmpi module file to load after the
compiler module and based on that select different paths based on the
currently-loaded compiler module.  If the user changes the compiler
module, will that cause the mpi module to also be reloaded so the new
settings will be loaded?  Or do I need this at all?

Thanks for all your help!
--Jim
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users