I can see the difference - we built Open MPI with tm support. For some
reason, I thought mpirun fed its environment to orted (after orted is
launched) so orted can pass it on to MPI tasks. That should be portable
between different launch mechanisms. But it looks like tm launches
orted with the full mpirun environment (at the request of mpirun).
Cheers,
David
Michael Sternberg wrote:
Hi David,
Hmm, your demo is well-chosen and crystal-clear, yet the output is unexpected.
I do not see environment vars passed by default here:
login3$ qsub -l nodes=2:ppn=1 -I
qsub: waiting for job 34683.mds01 to start
qsub: job 34683.mds01 ready
n102$ mpirun -n 2 -machinefile $PBS_NODEFILE hostname
n102
n085
n102$ mpirun -n 2 -machinefile $PBS_NODEFILE env | grep FOO
n102$ export FOO=BAR
n102$ mpirun -n 2 -machinefile $PBS_NODEFILE env | grep FOO
FOO=BAR
n102$ type mpirun
mpirun is hashed (/opt/soft/openmpi-1.3.2-intel10-1/bin/mpirun)
Curious, what do you get upon:
where mpirun
I built OpenMPI-1.3.2 here from source with:
CC=icc CXX=icpc FC=ifort F77=ifort \
LDFLAGS='-Wl,-z,noexecstack' \
CFLAGS='-O2 -g -fPIC' \
CXXFLAGS='-O2 -g -fPIC' \
FFLAGS='-O2 -g -fPIC' \
./configure --prefix=$prefix \
--with-libnuma=/usr \
--with-openib=/usr \
--with-udapl \
--enable-mpirun-prefix-by-default \
--without-tm
I did't find the behavior I saw strange, given that orterun(1) talks only about $OPMI_*
and inheritance from the remote shell. It also mentions a "boot MCA module",
about which I couldn't find much on open-mpi.org - hmm.
In the meantime, I did find a possible solution, namely, to tell ssh to pass a
variable using SendEnv/AcceptEnv. That variable is then seen by and can be
interpreted (cautiously) in /etc/profile.d/ scripts. A user could set it in
the job file (or even qalter it post submission):
#PBS -v VARNAME=foo:bar:baz
For VARNAME, I think simply "MODULES" or "EXTRAMODULES" could do.
With best regards,
Michael
On Nov 17, 2009, at 4:29 , David Singleton wrote:
I'm not sure why you dont see Open MPI behaving like other MPI's w.r.t.
modules/environment on remote MPI tasks - we do.
xe:~ > qsub -q express -lnodes=2:ppn=8,walltime=10:00,vmem=2gb -I
qsub: waiting for job 376366.xepbs to start
qsub: job 376366.xepbs ready
[dbs900@x27 ~]$ module load openmpi
[dbs900@x27 ~]$ mpirun -n 2 --bynode hostname
x27
x28
[dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep FOO
[dbs900@x27 ~]$ setenv FOO BAR
[dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep FOO
FOO=BAR
FOO=BAR
[dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep amber
[dbs900@x27 ~]$ module load amber
[dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep amber
LOADEDMODULES=openmpi/1.3.3:amber/9
PATH=/apps/openmpi/1.3.3/bin:/home/900/dbs900/bin:/bin:/usr/bin::/opt/bin:/usr/X11R6/bin:/opt/pbs/bin:/sbin:/usr/sbin:/apps/amber/9/exe
_LMFILES_=/apps/Modules/modulefiles/openmpi/1.3.3:/apps/Modules/modulefiles/amber/9
AMBERHOME=/apps/amber/9
LOADEDMODULES=openmpi/1.3.3:amber/9
PATH=/apps/openmpi/1.3.3/bin:/home/900/dbs900/bin:/bin:/usr/bin:/opt/bin:/usr/X11R6/bin:/opt/pbs/bin:/sbin:/usr/sbin:/apps/amber/9/exe
_LMFILES_=/apps/Modules/modulefiles/openmpi/1.3.3:/apps/Modules/modulefiles/amber/9
AMBERHOME=/apps/amber/9
David
Michael Sternberg wrote:
Dear readers,
With OpenMPI, how would one go about requesting to load environment modules (of
the http://modules.sourceforge.net/ kind) on remote nodes, augmenting those
normally loaded there by shell dotfiles?
Background:
I run a RHEL-5/CentOS-5 cluster. I load a bunch of default modules through
/etc/profile.d/ and recommend to users to customize modules in ~/.bashrc. A
problem arises for PBS jobs which might need job-specific modules, e.g., to
pick a specific flavor of an application. With other MPI implementations
(ahem) which export all (or judiciously nearly all) environment variables by
default, you can say:
#PBS ...
module load foo # not for OpenMPI
mpirun -np 42 ... \
bar-app
Not so with OpenMPI - any such customization is only effective for processes on
the master (=local) node of the job, and any variables changed by a given
module would have to be specifically passed via mpirun -x VARNAME. On the
remote nodes, those variables are not available in the dotfiles because they
are passed only once orted is live (after dotfile processing by the shell),
which then immediately spawns the application binaries (right?)
I thought along the following lines:
(1) I happen to run Lustre, which would allow writing a file coherently across
nodes prior to mpirun, and thus hook into the shell dotfile processing, but
that seems rather crude.
(2) "mpirun -x PATH -x LD_LIBRARY_PATH …" would take care of a lot, but is not
really general.
Is there a recommended way?
regards,
Michael
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
--------------------------------------------------------------------------
Dr David Singleton ANU Supercomputer Facility
HPC Systems Manager and NCI National Facility
david.single...@anu.edu.au Leonard Huxley Bldg (No. 56)
Phone: +61 2 6125 4389 Australian National University
Fax: +61 2 6125 8199 Canberra, ACT, 0200, Australia
--------------------------------------------------------------------------
