Re: [OMPI users] Suggestions on multi-compiler/multi-mpi build?
On Fri, 2007-11-30 at 19:05 -0800, Jim Kusznir wrote: > Thank you for your response! > > Just to clarify some things for my understanding: > > Do users load a single module that specifies both compiler and mpi > version (as opposed to loading two different modules, one for > complier, and one for mpi)? We combine compiler and MPI because it's necessary that the mpifcc/cxx/f90 be compatible with the compiler, so the correct compiler needs to be in the user's path, and it's just easier on the users to have them load a single module. A module can load another module, so if you create a module for, say, the PGI compilers, the appropriate OpenMPI module can load that module itself. Here's an example, my ompi-pgi script (note: all my scripts were cribbed from old ROCKS scripts; ROCKS doesn't include modules by default anymore but they used to provide it): #% Module3.0# ## ## CRAYfw mpich ## proc ModulesHelp { } { global version MPIhome puts stderr "\tCRAY freeware: OpenMPI\n" puts stderr "\tThis module loads the lastest versions of OpenMPI.\n" puts stderr "\n\tThis adds $OpenMPI/* to several of the" puts stderr "\tenvironment variables." puts stderr "\n\tVersion $version\n" } module-whatis "loads the OpenMPI environment for PGI over Ethernet" # for Tcl script use only set version 1.2 set MPIhome /opt/openmpi-1.2/ethernet/pgi prepend-pathPATH$MPIhome/bin prepend-pathINCLUDE $MPIhome/include prepend-pathINFOPATH$MPIhome/share prepend-pathLD_LIBRARY_PATH $MPIhome/lib setenv OMPI_MPIBIN $MPIhome setenv OMPI_MPIHOME $MPIhome setenv OMPI_MPIF77 "pgf77" setenv OMPI_MPIF90 "pgf90" setenv OMPI_MPICC "pgcc" setenv OMPI_MPICXX "pgCC" if [ module-info mode load ] { if {! [ is-loaded pgi ] } { module load pgi } if { [ is-loaded ompi-pgi ] } { puts stderr "\tModule ompi-pgi already loaded\n" } if { [ is-loaded ompi-gnu ] } { puts stderr "\tModule ompi-gnu superceded\n" } if { [ is-loaded ompi-intel ] } { puts stderr "\tModule ompi-intel superceded\n" } if { [ is-loaded mpich-eth-pgi ] } { puts stderr "\tModule mpich-eth-pgi environment superceded.\n" } if { [ is-loaded mpich-eth-gnu ] } { puts stderr "\tModule mpich-eth-gnu environment superceded.\n" } if { [ is-loaded mpich-eth-intel ] } { puts stderr "\tModule mpich-eth-intel environment superceded.\n" } } > > So, if the MPI implementations need to be built separately for each > compiler, what about other software packages that users' link against? > For example, as far as I know, there are not seperate versions of > libc for each compiler (but I may be ignorant hereplease correct > me if I'm wrong). How about other packages that code #includes? How > do I know when something needs to be compiled separately vs one > version for everyone? We build a distinct version at least for every library that has F90/95 bindings. Probably a good idea for C++ as well. > > Another thing that I'm still not quite clear on: Can I have all > versions of the various MPI libraries in the ld path at once (so the > user doesn't have to "select" the correct one somehow prior to code > execution)? I'll let somebody who is more of a real system administrator answer that. > > Are there any tricks to building/installing all this stuff in a fairly > automated manner? > We use ROCKS. You have to create an rpm for everything, but once you do then propagating it is automated. -- Katherine Holcomb, Ph.D.kholc...@virginia.edu Research Computing Support Group - ITC Office Phone: (434) 982-5948 I046 BSEL, Clark HallCenter Phone: (434) 243-8799 University of Virginia 22904
Re: [OMPI users] Suggestions on multi-compiler/multi-mpi build?
We have almost exactly the situation you describe with our clusters. I'm not the system administrator but I am the one who actually writes the module scripts. It is best to compile OpenMPI (and any other MPI) with each compiler separately; this is especially necessary for the Fortran and C++ bindings. We simply have a directory layout like /opt/openmpi/intel /opt/openmpi/pgi /opt/openmpi/gnu etc. To compile OpenMPI with a different compiler, you can use environment variables such as (for the Intel compiler) FC=ifort F90=ifort CC=icc CXX=icpc You set these within the %build part of the rpm spec file. Similarly for MPICH2. (I would recommend MPICH2 over MPICH1 at this point.) Users load a module to select a compiler+MPI combination. Loading the appropriate module sets the path for the version of mpicc/cxx/f90 chosen by the user. That in turn knows where its associated libraries live. Most users link MPI libraries statically. If dynamic libraries are needed then they can be added to LD_LIBRARY_PATH by the module script. I am away from my office until after the Thanksgiving holiday but if you email me personally then, I can send you some sample module scripts. I can also send you some sample spec files for the rpms we use. - Original Message - From: "Jim Kusznir"To: Sent: Thursday, November 15, 2007 11:54 AM Subject: [OMPI users] Suggestions on multi-compiler/multi-mpi build? Hi all: I'm trying to set up a cluster for a group of users with very different needs. So far, it looks like I need gcc, pgi, and intel to work with openmpi and mpich, with each user able to control what combination they get. This is turning out to be much more difficult than I expected. Someone has pointed me to enviornment-modules ("Modules"), which looks like it will be a critical part of the solution. I even noticed that the provided openmpi.spec file has some direct support for modules, which makes me quite happy! However, I still have many questions about how to set things up. First, I get the impression that openmpi will need to be compiled with each compiler that will use it. If this is true, I'm not quite sure how to go about it. I could install in different directories for the user commands, but what about the libraries? I don't think I have a feesable way of selecting which library to use on the fly on the entire cluster for each user, so it seems like it would be better to have all the libraries available. In addition, I will need RPMs to deploy efficiently on the cluster. I suspect I can change the versioning info and build with each compiler, but at this point, I don't even know how to reliably select what compiler rpmbuild will use (I've only succeeded in using gcc). Finally, using modules, how do I set it up so that if a user changes compilers, but stays with openmpi, it will load the correct openmpi paths? I know I can set up the openmpi module file to load after the compiler module and based on that select different paths based on the currently-loaded compiler module. If the user changes the compiler module, will that cause the mpi module to also be reloaded so the new settings will be loaded? Or do I need this at all? Thanks for all your help! --Jim ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] Any known issues with ksh?
It reports itself as version 1.1.1 On Tue, 2006-12-05 at 10:15 -0700, Ralph H Castain wrote: > Hi Katherine > > Could you tell us which version of OpenMPI you are using? > > Thanks > Ralph > > > On 12/5/06 10:02 AM, "Katherine Holcomb" <kholc...@virginia.edu> wrote: > > > We are in the process of converting our clusters from MPICH to OpenMPI > > and have encountered some odd problems. For historical reasons, the > > default shell we install for users is ksh. The --profile option to > > mpirun does not work for our users under ksh. We are unsure why > > although our system administrator believes that OpenMPI may be making > > the ksh on the remote nodes believe that it's a login shell. We've > > tried both rsh and ssh. Has anybody seen anything like this? > > > > There are no problems at all under bash. (I use bash and I had no > > trouble, but then started switching some users who were having MPICH > > problems and they could not get it to work unless they changed to bash.) > > > _______ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Katherine Holcomb, Ph.D.kholc...@virginia.edu Research Computing Support Group - ITC Office Phone: (434) 982-5948 148 BSEL, Clark HallCenter Phone: (434) 243-8799 University of Virginia 22904
[OMPI users] Any known issues with ksh?
We are in the process of converting our clusters from MPICH to OpenMPI and have encountered some odd problems. For historical reasons, the default shell we install for users is ksh. The --profile option to mpirun does not work for our users under ksh. We are unsure why although our system administrator believes that OpenMPI may be making the ksh on the remote nodes believe that it's a login shell. We've tried both rsh and ssh. Has anybody seen anything like this? There are no problems at all under bash. (I use bash and I had no trouble, but then started switching some users who were having MPICH problems and they could not get it to work unless they changed to bash.) -- Katherine Holcomb, Ph.D.kholc...@virginia.edu Research Computing Support Group - ITC Office Phone: (434) 982-5948 148 BSEL, Clark HallCenter Phone: (434) 243-8799 University of Virginia 22904
Re: [OMPI users] Starting on remote nodes
> > I presume you left some critical piece of information out of your > message, like the name and configuration of the batch queueing system > you are using. We're using PBS Pro although I don't think it's a factor in this particular situation. (I did find some behavior with PBS Pro that seemed not to be as advertised, i.e. it was placing all the processes on one node when two were requested unless the -machinefile flag was explicitly set to $PBS_NODEFILE, but that was a different problem.) > > The answer to your question as worded may not be the best answer for > your problem. > > I have dealt with two cases similar to yours: > > 1) Large system using Modules and LSF batch queueing system -- this > type of system requires the people configuring LSF to set up some > stuff or the end users have to use --prefix flag to get the OpenMPI > path, plus more to get the correct compiler (something I never > figured out how to do before the LSF admins extended their LSF > installation to cover OpenMPI). [what stuff I don't know, I'm not an > LSF admin] It does look like we'll have to use the --prefix flag, at least to start. Rainer Keller pointed out that I can set an environment variable in the module script and that does seem to be the best option for now. We'd rather not get into wrapping the binaries. > > 2) Local system I sysadm, learning Modules setup was going to take > more time than I had available so I wrote a script that sets PATH, > MANPATH, and LD_LIBRARY_PATH based on similar arguments as the real > Module software (also G95_INCLUDE_PATH for g95). When the user sets > the environmental variables via my script and then runs OpenMPI I see > no problems with OpenMPI on the other nodes; however, we don't have a > batch queuing system. I don't see why using the Modules software > would be any different. One critical piece is that my script also > aliases mpirun, for example "alias mpirun "mpirun --prefix /opt/g95/ > openmpi/1.1.1 " (which the real modules software should also be able > to do if needed) and I have only one installation of each type of > compiler (g95, Intel, PGI, Absoft). Long term we are probably going to do something similar (write our own Modules replacement). For one thing, the Modules software doesn't seem to have been maintained for a while, and for another, it uses Tcl, which is not much of a mainstream language anymore. -- Katherine Holcomb, Ph.D.kholc...@virginia.edu Research Computing Support Group - ITC Office Phone: (434) 982-5948 148 BSEL, Clark HallCenter Phone: (434) 243-8799 University of Virginia 22904
[OMPI users] Starting on remote nodes
We currently use MPICH on our Linux clusters, but due to a high frequency of semaphore problems we are planning to replace it. OpenMPI looks like our best candidate but we have hit a snag. We support multiple compilers (specifically PGI and Intel) and due to incompatibilities in different vendors' f90 .mod files, we have separate directories for OpenMPI with each compiler. Therefore we cannot set a global path to the OpenMPI binaries -- it will differ depending on the user's choice of compiler. I have read about the --prefix flag and this does work, but our users are mostly barely conversant with Unix and many would have difficulty finding and specifying the appropriate path. (We use the modules software environment currently to set paths and the like for them.) Is there some way to specify something like "use the same path as you are in" from the root process? There was some allusion in the FAQ to changing the wrappers to include directives, but the link led to a "no such category" page. -- Katherine Holcomb, Ph.D.kholc...@virginia.edu Research Computing Support Group - ITC Office Phone: (434) 982-5948 148 BSEL, Clark HallCenter Phone: (434) 243-8799 University of Virginia 22904