Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Gus Correa]

Jeffrey A Cummings wrote:
Thanks for all the good replies on this thread.  I don't know if I'll be 
able to make a dent in the corporate IT bureaucracy but I'm going to try.
 




From:Prentice Bisbal 
To:Open MPI Users 
Date:02/02/2011 11:35 AM
Subject:Re: [OMPI users] How closely tied is a specific release 
of OpenMPI to the host operating system and other system software?

Sent by:users-boun...@open-mpi.org




Jeffrey A Cummings wrote:
 > I use OpenMPI on a variety of platforms:  stand-alone servers running
 > Solaris on sparc boxes and Linux (mostly CentOS) on AMD/Intel boxes,
 > also Linux (again CentOS) on large clusters of AMD/Intel boxes.  These
 > platforms all have some version of the 1.3 OpenMPI stream.  I recently
 > requested an upgrade on all systems to 1.4.3 (for production work) and
 > 1.5.1 (for experimentation).  I'm getting a lot of push back from the
 > SysAdmin folks claiming that OpenMPI is closely intertwined with the
 > specific version of the operating system and/or other system software
 > (i.e., Rocks on the clusters).  I need to know if they are telling me
 > the truth or if they're just making excuses to avoid the work.  To state
 > my question another way:  Apparently each release of Linux and/or Rocks
 > comes with some version of OpenMPI bundled in.  Is it dangerous in some
 > way to upgrade to a newer version of OpenMPI?  Thanks in advance for any
 > insight anyone can provide.
 >
 > - Jeff
 >

Jeff,

OpenMPI is more or less a user-space program, and isn't that tightly
coupled to the OS at all. As long as the OS has the correct network
drivers (ethernet, IB, or other), that's all OpenMPI needs to do it's
job. In fact, you can install it yourself in your own home directory (if
your home directory is shared amongst the cluster nodes you want to
use), and run it from there - no special privileges needed.

I have many different versions of OpenMPI installed on my systems,
without a problem.

As a system administrator responsible for maintaining OpenMPI on several
clusters, it sounds like one of two things:

1. Your system administrators really don't know what they're talking
about, or,

2. They're lying to you to avoid doing work.

--
Prentice


Jeff

Worst scenario, you can install OpenMPI yourself, from the source 
tarball, in a subdirectory of your ${HOME}, for instance.

You would just need adjust your PATH and LD_LIBRARY_PATH.

Gus Correa

Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Joe Landman]

On 2/1/2011 5:02 PM, Jeffrey A Cummings wrote:

I use OpenMPI on a variety of platforms: stand-alone servers running
Solaris on sparc boxes and Linux (mostly CentOS) on AMD/Intel boxes,
also Linux (again CentOS) on large clusters of AMD/Intel boxes. These
platforms all have some version of the 1.3 OpenMPI stream. I recently
requested an upgrade on all systems to 1.4.3 (for production work) and
1.5.1 (for experimentation). I'm getting a lot of push back from the
SysAdmin folks claiming that OpenMPI is closely intertwined with the
specific version of the operating system and/or other system software
(i.e., Rocks on the clusters). I need to know if they are telling me the
truth or if they're just making excuses to avoid the work. To state my
question another way: Apparently each release of Linux and/or Rocks
comes with some version of OpenMPI bundled in. Is it dangerous in some
way to upgrade to a newer version of OpenMPI? Thanks in advance for any
insight anyone can provide.


Jeff:

  The issue is that the way Rocks integrates MPI stacks, it is quite 
hard to upgrade the baseline stacks.


  This said, there is no issue in placing new stacks in 
/shared/openmpi/version/compiler/  or similar.  We've done this for many 
of our Rocks customers.


  For Rocks, the admins usually need RPMs, then they have to work on 
some extend-compute.xml magic to make it work.  You can do this, or as I 
did when we ran on a shared Rocks cluster that the admin was ... er ... 
reluctant ... to make needed changes.  You can easily install it to your 
home directory.  No admin needed for this.  Worked nicely.  Solved our 
problems.


Regards,

Joe

--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.,
email: land...@scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615

Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Jeffrey A Cummings]

Thanks for all the good replies on this thread.  I don't know if I'll be 
able to make a dent in the corporate IT bureaucracy but I'm going to try.




From:   Prentice Bisbal 
To: Open MPI Users 
List-Post: users@lists.open-mpi.org
Date:   02/02/2011 11:35 AM
Subject:Re: [OMPI users] How closely tied is a specific release of 
OpenMPI to the host operating system and other system software?
Sent by:users-boun...@open-mpi.org



Jeffrey A Cummings wrote:
> I use OpenMPI on a variety of platforms:  stand-alone servers running
> Solaris on sparc boxes and Linux (mostly CentOS) on AMD/Intel boxes,
> also Linux (again CentOS) on large clusters of AMD/Intel boxes.  These
> platforms all have some version of the 1.3 OpenMPI stream.  I recently
> requested an upgrade on all systems to 1.4.3 (for production work) and
> 1.5.1 (for experimentation).  I'm getting a lot of push back from the
> SysAdmin folks claiming that OpenMPI is closely intertwined with the
> specific version of the operating system and/or other system software
> (i.e., Rocks on the clusters).  I need to know if they are telling me
> the truth or if they're just making excuses to avoid the work.  To state
> my question another way:  Apparently each release of Linux and/or Rocks
> comes with some version of OpenMPI bundled in.  Is it dangerous in some
> way to upgrade to a newer version of OpenMPI?  Thanks in advance for any
> insight anyone can provide.
> 
> - Jeff
> 

Jeff,

OpenMPI is more or less a user-space program, and isn't that tightly
coupled to the OS at all. As long as the OS has the correct network
drivers (ethernet, IB, or other), that's all OpenMPI needs to do it's
job. In fact, you can install it yourself in your own home directory (if
 your home directory is shared amongst the cluster nodes you want to
use), and run it from there - no special privileges needed.

I have many different versions of OpenMPI installed on my systems,
without a problem.

As a system administrator responsible for maintaining OpenMPI on several
clusters, it sounds like one of two things:

1. Your system administrators really don't know what they're talking
about, or,

2. They're lying to you to avoid doing work.

--
Prentice
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Prentice Bisbal]

Jeffrey A Cummings wrote:
> I use OpenMPI on a variety of platforms:  stand-alone servers running
> Solaris on sparc boxes and Linux (mostly CentOS) on AMD/Intel boxes,
> also Linux (again CentOS) on large clusters of AMD/Intel boxes.  These
> platforms all have some version of the 1.3 OpenMPI stream.  I recently
> requested an upgrade on all systems to 1.4.3 (for production work) and
> 1.5.1 (for experimentation).  I'm getting a lot of push back from the
> SysAdmin folks claiming that OpenMPI is closely intertwined with the
> specific version of the operating system and/or other system software
> (i.e., Rocks on the clusters).  I need to know if they are telling me
> the truth or if they're just making excuses to avoid the work.  To state
> my question another way:  Apparently each release of Linux and/or Rocks
> comes with some version of OpenMPI bundled in.  Is it dangerous in some
> way to upgrade to a newer version of OpenMPI?  Thanks in advance for any
> insight anyone can provide.
> 
> - Jeff
> 

Jeff,

OpenMPI is more or less a user-space program, and isn't that tightly
coupled to the OS at all. As long as the OS has the correct network
drivers (ethernet, IB, or other), that's all OpenMPI needs to do it's
job. In fact, you can install it yourself in your own home directory (if
 your home directory is shared amongst the cluster nodes you want to
use), and run it from there - no special privileges needed.

I have many different versions of OpenMPI installed on my systems,
without a problem.

As a system administrator responsible for maintaining OpenMPI on several
clusters, it sounds like one of two things:

1. Your system administrators really don't know what they're talking
about, or,

2. They're lying to you to avoid doing work.

--
Prentice

Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Jeff Squyres]

On Feb 2, 2011, at 7:02 AM, Terry Dontje wrote:

> 2.  The system libraries on different linux versions are not always the same. 
>  At Oracle we build a binary distribution of OMPI that we test out on several 
> different versions of Linux.  The key here is building on a machine that is 
> essentially the lowest common denominator of all the system software that 
> exists on the machines one will be running on.  This is essentially why 
> Oracle states a bounded set of OS versions a distribution runs on.  An 
> example of this is there is a component in OMPI that was relying on a version 
> of libbfd that changed significantly between Linux version.  Once we got rid 
> of the usage of that library we were ok.  There are not "a lot" of these 
> instances but the number is not zero.  

+1

If your systems are not running exactly the same versions of software (e.g., 
different OS's), we typically categorize this as a "heterogeneous" scenario -- 
with all the caveats that Terry mentions.  You may well likely require a 
separate OMPI install for each OS, and possibly (likely) even a different 
compilation of your application for each OS.

In general, I try to limit my OMPI jobs to homogeneous platforms.  
Heterogeneous can be done; it's just somewhat of a pain to get right.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Terry Dontje]

On 02/01/2011 07:34 PM, Jeff Squyres wrote:

On Feb 1, 2011, at 5:02 PM, Jeffrey A Cummings wrote:


I'm getting a lot of push back from the SysAdmin folks claiming that OpenMPI is 
closely intertwined with the specific version of the operating system and/or 
other system software (i.e., Rocks on the clusters).

I wouldn't say that this is true.  We test across a wide variety of OS's and 
compilers.  I'm sure that there are particular platforms/environments that can 
trip up some kind of problem (it's happened before), but in general, Open MPI 
is pretty portable.


To state my question another way:  Apparently each release of Linux and/or 
Rocks comes with some version of OpenMPI bundled in.  Is it dangerous in some 
way to upgrade to a newer version of OpenMPI?

Not at all.  Others have said it, but I'm one of the developers and I'll 
reinforce their answers: I regularly have about a dozen different installations 
of Open MPI on my cluster at any given time (all in different stages of 
development -- all installed to different prefixes).  I switch between them 
quite easily by changing my PATH and LD_LIBRARY_PATH (both locally and on 
remote nodes).
Not to be a lone descenting opinion here is my experience in doing the 
above.


First if you are always recompiling your application with a specific 
version of OMPI then I would agree with everything Jeff said above.  
That is you can build many versions of OMPI on many linux versions and 
have them run.


But there are definite pitfalls once you start trying to keep one set of 
executables and OMPI binaries across different Linux versions.


1.  You may see executables not be able to use OMPI libraries that 
differ in the first dot number release (eg 1.3 vs 1.4 or 1.5 branches).  
We the community try to avoid these incompatibilities as much as 
possible but it happens on occasion (I think 1.3 to 1.4 is one such 
occasion).


2.  The system libraries on different linux versions are not always the 
same.  At Oracle we build a binary distribution of OMPI that we test out 
on several different versions of Linux.  The key here is building on a 
machine that is essentially the lowest common denominator of all the 
system software that exists on the machines one will be running on.  
This is essentially why Oracle states a bounded set of OS versions a 
distribution runs on.  An example of this is there is a component in 
OMPI that was relying on a version of libbfd that changed significantly 
between Linux version.  Once we got rid of the usage of that library we 
were ok.  There are not "a lot" of these instances but the number is not 
zero.


--
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle *- Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com 

Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Jeff Squyres]

On Feb 1, 2011, at 5:02 PM, Jeffrey A Cummings wrote:

> I'm getting a lot of push back from the SysAdmin folks claiming that OpenMPI 
> is closely intertwined with the specific version of the operating system 
> and/or other system software (i.e., Rocks on the clusters).  

I wouldn't say that this is true.  We test across a wide variety of OS's and 
compilers.  I'm sure that there are particular platforms/environments that can 
trip up some kind of problem (it's happened before), but in general, Open MPI 
is pretty portable.

> To state my question another way:  Apparently each release of Linux and/or 
> Rocks comes with some version of OpenMPI bundled in.  Is it dangerous in some 
> way to upgrade to a newer version of OpenMPI?  

Not at all.  Others have said it, but I'm one of the developers and I'll 
reinforce their answers: I regularly have about a dozen different installations 
of Open MPI on my cluster at any given time (all in different stages of 
development -- all installed to different prefixes).  I switch between them 
quite easily by changing my PATH and LD_LIBRARY_PATH (both locally and on 
remote nodes).

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Gustavo Correa]

On Feb 1, 2011, at 5:02 PM, Jeffrey A Cummings wrote:

> I use OpenMPI on a variety of platforms:  stand-alone servers running Solaris 
> on sparc boxes and Linux (mostly CentOS) on AMD/Intel boxes, also Linux 
> (again CentOS) on large clusters of AMD/Intel boxes.  These platforms all 
> have some version of the 1.3 OpenMPI stream.  I recently requested an upgrade 
> on all systems to 1.4.3 (for production work) and 1.5.1 (for 
> experimentation).  I'm getting a lot of push back from the SysAdmin folks 
> claiming that OpenMPI is closely intertwined with the specific version of the 
> operating system and/or other system software (i.e., Rocks on the clusters).  
> I need to know if they are telling me the truth or if they're just making 
> excuses to avoid the work.  To state my question another way:  Apparently 
> each release of Linux and/or Rocks comes with some version of OpenMPI bundled 
> in.  Is it dangerous in some way to upgrade to a newer version of OpenMPI?  
> Thanks in advance for any insight anyone can provide. 
> 
> - Jeff___

Hi Jeffrey

As others said, Rocks has a default MPI (some version OpenMPI built with Gnu 
compilers with
support for Ethernet only) which comes with the "hpc" Rocks roll.
You can use that MPI, but you don't have to.

This doesn't prevent you to install any other version of OpenMPI (actually of 
any other software)
with support to whatever you have (e.g. Infiniband, Torque resource manager, 
using other compilers than Gnu, etc).

The right location to install on Rocks is the /share/apps directory of the 
head/frontend node,
which is NFS mounted on the nodes.
It is wise to use subdirectories with names identifying your version somehow,
e.g. /share/apps/ompi-1.4.3/intel-11.1.020, for something compiled with intel 
compilers.

The --prefix=/share/apps/bla/bla option of OpenMPI configure will put the 
installed directory tree wherever you want.

'configure --help' will tell tons of possibilities (e.g. tight coupling with 
Torque os SGE,
Infinband support, etc).

You also need to set the user environment.

A simple minded way is to prepend the OpenMPI bin directory to the PATH
environment variable (say in the .bashrc/.cshrc user file), and the lib 
directory
to the LD_LIBRARY_PATH.
Adding share/man to the MANPATH is not mandatory, but helpful.
This is rather inflexible and requires editing those initialization files every 
time you
want to switch the MPI version you use, though.

A much better and flexible way, as was also mentioned, is to use the 
environment modules,
but your Sys Admin must be willing to learn how to write the corresponding 
module files
(in Tcl/Tk jargon).
This will allow you switch to across different versions by just issuing a 
command line
like 'module switch path/to/old/version  path/to/new/version'.

See:
http://modules.sourceforge.net/

I can't speak about Solaris, but it also supports environment modules, if I am 
not mistaken.

I hope this helps,
Gus Correa

Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Reuti]

Am 01.02.2011 um 23:02 schrieb Jeffrey A Cummings:

> I use OpenMPI on a variety of platforms:  stand-alone servers running Solaris 
> on sparc boxes and Linux (mostly CentOS) on AMD/Intel boxes, also Linux 
> (again CentOS) on large clusters of AMD/Intel boxes.  These platforms all 
> have some version of the 1.3 OpenMPI stream.  I recently requested an upgrade 
> on all systems to 1.4.3 (for production work) and 1.5.1 (for 
> experimentation).  I'm getting a lot of push back from the SysAdmin folks 
> claiming that OpenMPI is closely intertwined with the specific version of the 
> operating system and/or other system software (i.e., Rocks on the clusters).  
> I need to know if they are telling me the truth or if they're just making 
> excuses to avoid the work.

Maybe ROCKS or whatever provides only one version. Anyway: you can download 
Open MPI, compile it to build into e.g. ~/local/openmpi-1.4.3, adjust your 
PATHs and you are done.

Unless you build it with static libraries, it might in addition be necessary to 
adjust LD_LIBRARY_PATH at runtime.

I use most often my own version on the clusters I have access to and disregard 
any installed one.

-- Reuti


>  To state my question another way:  Apparently each release of Linux and/or 
> Rocks comes with some version of OpenMPI bundled in.  Is it dangerous in some 
> way to upgrade to a newer version of OpenMPI?  Thanks in advance for any 
> insight anyone can provide. 
> 
> - Jeff___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Doug Reeder]

Jeff,

We have similar circumstances and have been able to install and use versions of 
openmpi newer than supplied with the OS. It is necessary  to have some means of 
path management to ensure that applications build against the desired version 
of openmpi and run with the version of openmpi they were built with. We use the 
module system for this path management. We create modules for each version of 
openmpi and each version of the applications. We than include the appropriate 
openmpi module in the module for the application. Then when a user loads a 
module for their application they automatically get the correct version of 
openmpi.

Doug Reeder
On Feb 1, 2011, at 2:02 PM, Jeffrey A Cummings wrote:

> I use OpenMPI on a variety of platforms:  stand-alone servers running Solaris 
> on sparc boxes and Linux (mostly CentOS) on AMD/Intel boxes, also Linux 
> (again CentOS) on large clusters of AMD/Intel boxes.  These platforms all 
> have some version of the 1.3 OpenMPI stream.  I recently requested an upgrade 
> on all systems to 1.4.3 (for production work) and 1.5.1 (for 
> experimentation).  I'm getting a lot of push back from the SysAdmin folks 
> claiming that OpenMPI is closely intertwined with the specific version of the 
> operating system and/or other system software (i.e., Rocks on the clusters).  
> I need to know if they are telling me the truth or if they're just making 
> excuses to avoid the work.  To state my question another way:  Apparently 
> each release of Linux and/or Rocks comes with some version of OpenMPI bundled 
> in.  Is it dangerous in some way to upgrade to a newer version of OpenMPI?  
> Thanks in advance for any insight anyone can provide. 
> 
> - Jeff___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Richard Walsh]

Jeff,

We have 3 Rocks Clusters, while there is a default MPI with each
Rocks Release, it is often behind the latest production release as
you note.

We typically install whatever OpenMPI version we want in a shared space
and ignore the default installed with Rocks.  Sometimes there standard
Linux libraries that can be a bit out of date which may be registered as
"can't finds" in the configuration and/or buiild of OpenMPI, but there usually 
an
easy go around.  As far as 'closely intertwined' goes, I would say that
is an exaggeration.

It does mean some extra work for someone ... around here is it me ... ;-) ...

rbw

Richard Walsh
Parallel Applications and Systems Manager
CUNY HPC Center, Staten Island, NY
718-982-3319
612-382-4620

Reason does give the heart pause;
As the heart gives reason fits.

Yet, to live where reason always rules;
Is to kill one's heart with wits.

From: users-boun...@open-mpi.org [users-boun...@open-mpi.org] on behalf of 
Jeffrey A Cummings [jeffrey.a.cummi...@aero.org]
Sent: Tuesday, February 01, 2011 5:02 PM
To: us...@open-mpi.org
Subject: [OMPI users] How closely tied is a specific release of OpenMPI to the 
host operating system and other system software?

I use OpenMPI on a variety of platforms:  stand-alone servers running Solaris 
on sparc boxes and Linux (mostly CentOS) on AMD/Intel boxes, also Linux (again 
CentOS) on large clusters of AMD/Intel boxes.  These platforms all have some 
version of the 1.3 OpenMPI stream.  I recently requested an upgrade on all 
systems to 1.4.3 (for production work) and 1.5.1 (for experimentation).  I'm 
getting a lot of push back from the SysAdmin folks claiming that OpenMPI is 
closely intertwined with the specific version of the operating system and/or 
other system software (i.e., Rocks on the clusters).  I need to know if they 
are telling me the truth or if they're just making excuses to avoid the work.  
To state my question another way:  Apparently each release of Linux and/or 
Rocks comes with some version of OpenMPI bundled in.  Is it dangerous in some 
way to upgrade to a newer version of OpenMPI?  Thanks in advance for any 
insight anyone can provide.

- Jeff



Think green before you print this email.

[OMPI users] How closely tied is a specific release of OpenMPI to the host operating system and other system software?

[Jeffrey A Cummings]

I use OpenMPI on a variety of platforms:  stand-alone servers running 
Solaris on sparc boxes and Linux (mostly CentOS) on AMD/Intel boxes, also 
Linux (again CentOS) on large clusters of AMD/Intel boxes.  These 
platforms all have some version of the 1.3 OpenMPI stream.  I recently 
requested an upgrade on all systems to 1.4.3 (for production work) and 
1.5.1 (for experimentation).  I'm getting a lot of push back from the 
SysAdmin folks claiming that OpenMPI is closely intertwined with the 
specific version of the operating system and/or other system software 
(i.e., Rocks on the clusters).  I need to know if they are telling me the 
truth or if they're just making excuses to avoid the work.  To state my 
question another way:  Apparently each release of Linux and/or Rocks comes 
with some version of OpenMPI bundled in.  Is it dangerous in some way to 
upgrade to a newer version of OpenMPI?  Thanks in advance for any insight 
anyone can provide.

- Jeff