Not exactly. It completely depends on how Torque was setup - OMPI isn't forwarding the environment. Torque is.
We made a design decision at the very beginning of the OMPI project not to forward non-OMPI envars unless directed to do so by the user. I'm afraid I disagree with Michael's claim that other MPIs do forward them - yes, MPICH does, but not all others do. The world is bigger than MPICH and OMPI :-) Since there is inconsistency in this regard between MPIs, we chose not to forward. Reason was simple: there is no way to know what is safe to forward vs what is not (e.g., what to do with DISPLAY), nor what the underlying environment is trying to forward vs what it isn't. It is very easy to get cross-wise and cause totally unexpected behavior, as users have complained about for years. First, if you are using a managed environment like Torque, we recommend that you work with your sys admin to decide how to configure it. This is the best way to resolve a problem. Second, if you are not using a managed environment and/or decide not to have that environment do the forwarding, you can tell OMPI to forward the envars you need by specifying them via the -x cmd line option. We already have a request to expand this capability, and I will be doing so as time permits. One option I'll be adding is the reverse of -x - i.e., "forward all envars -except- the specified one(s)". HTH ralph On Nov 17, 2009, at 5:55 AM, David Singleton wrote: > > I can see the difference - we built Open MPI with tm support. For some > reason, I thought mpirun fed its environment to orted (after orted is > launched) so orted can pass it on to MPI tasks. That should be portable > between different launch mechanisms. But it looks like tm launches > orted with the full mpirun environment (at the request of mpirun). > > Cheers, > David > > > Michael Sternberg wrote: >> Hi David, >> Hmm, your demo is well-chosen and crystal-clear, yet the output is >> unexpected. I do not see environment vars passed by default here: >> login3$ qsub -l nodes=2:ppn=1 -I >> qsub: waiting for job 34683.mds01 to start >> qsub: job 34683.mds01 ready >> n102$ mpirun -n 2 -machinefile $PBS_NODEFILE hostname >> n102 >> n085 >> n102$ mpirun -n 2 -machinefile $PBS_NODEFILE env | grep FOO >> n102$ export FOO=BAR >> n102$ mpirun -n 2 -machinefile $PBS_NODEFILE env | grep FOO >> FOO=BAR >> n102$ type mpirun >> mpirun is hashed (/opt/soft/openmpi-1.3.2-intel10-1/bin/mpirun) >> Curious, what do you get upon: >> where mpirun >> I built OpenMPI-1.3.2 here from source with: >> CC=icc CXX=icpc FC=ifort F77=ifort \ >> LDFLAGS='-Wl,-z,noexecstack' \ >> CFLAGS='-O2 -g -fPIC' \ >> CXXFLAGS='-O2 -g -fPIC' \ >> FFLAGS='-O2 -g -fPIC' \ >> ./configure --prefix=$prefix \ >> --with-libnuma=/usr \ >> --with-openib=/usr \ >> --with-udapl \ >> --enable-mpirun-prefix-by-default \ >> --without-tm >> I did't find the behavior I saw strange, given that orterun(1) talks only >> about $OPMI_* and inheritance from the remote shell. It also mentions a >> "boot MCA module", about which I couldn't find much on open-mpi.org - hmm. >> In the meantime, I did find a possible solution, namely, to tell ssh to pass >> a variable using SendEnv/AcceptEnv. That variable is then seen by and can >> be interpreted (cautiously) in /etc/profile.d/ scripts. A user could set it >> in the job file (or even qalter it post submission): >> #PBS -v VARNAME=foo:bar:baz >> For VARNAME, I think simply "MODULES" or "EXTRAMODULES" could do. >> With best regards, >> Michael >> On Nov 17, 2009, at 4:29 , David Singleton wrote: >>> I'm not sure why you dont see Open MPI behaving like other MPI's w.r.t. >>> modules/environment on remote MPI tasks - we do. >>> >>> xe:~ > qsub -q express -lnodes=2:ppn=8,walltime=10:00,vmem=2gb -I >>> qsub: waiting for job 376366.xepbs to start >>> qsub: job 376366.xepbs ready >>> >>> [dbs900@x27 ~]$ module load openmpi >>> [dbs900@x27 ~]$ mpirun -n 2 --bynode hostname >>> x27 >>> x28 >>> [dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep FOO >>> [dbs900@x27 ~]$ setenv FOO BAR >>> [dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep FOO >>> FOO=BAR >>> FOO=BAR >>> [dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep amber >>> [dbs900@x27 ~]$ module load amber >>> [dbs900@x27 ~]$ mpirun -n 2 --bynode env | grep amber >>> LOADEDMODULES=openmpi/1.3.3:amber/9 >>> PATH=/apps/openmpi/1.3.3/bin:/home/900/dbs900/bin:/bin:/usr/bin::/opt/bin:/usr/X11R6/bin:/opt/pbs/bin:/sbin:/usr/sbin:/apps/amber/9/exe >>> _LMFILES_=/apps/Modules/modulefiles/openmpi/1.3.3:/apps/Modules/modulefiles/amber/9 >>> AMBERHOME=/apps/amber/9 >>> LOADEDMODULES=openmpi/1.3.3:amber/9 >>> PATH=/apps/openmpi/1.3.3/bin:/home/900/dbs900/bin:/bin:/usr/bin:/opt/bin:/usr/X11R6/bin:/opt/pbs/bin:/sbin:/usr/sbin:/apps/amber/9/exe >>> _LMFILES_=/apps/Modules/modulefiles/openmpi/1.3.3:/apps/Modules/modulefiles/amber/9 >>> AMBERHOME=/apps/amber/9 >>> >>> David >>> >>> >>> Michael Sternberg wrote: >>>> Dear readers, >>>> With OpenMPI, how would one go about requesting to load environment >>>> modules (of the http://modules.sourceforge.net/ kind) on remote nodes, >>>> augmenting those normally loaded there by shell dotfiles? >>>> Background: >>>> I run a RHEL-5/CentOS-5 cluster. I load a bunch of default modules >>>> through /etc/profile.d/ and recommend to users to customize modules in >>>> ~/.bashrc. A problem arises for PBS jobs which might need job-specific >>>> modules, e.g., to pick a specific flavor of an application. With other >>>> MPI implementations (ahem) which export all (or judiciously nearly all) >>>> environment variables by default, you can say: >>>> #PBS ... >>>> module load foo # not for OpenMPI >>>> mpirun -np 42 ... \ >>>> bar-app >>>> Not so with OpenMPI - any such customization is only effective for >>>> processes on the master (=local) node of the job, and any variables >>>> changed by a given module would have to be specifically passed via mpirun >>>> -x VARNAME. On the remote nodes, those variables are not available in >>>> the dotfiles because they are passed only once orted is live (after >>>> dotfile processing by the shell), which then immediately spawns the >>>> application binaries (right?) >>>> I thought along the following lines: >>>> (1) I happen to run Lustre, which would allow writing a file coherently >>>> across nodes prior to mpirun, and thus hook into the shell dotfile >>>> processing, but that seems rather crude. >>>> (2) "mpirun -x PATH -x LD_LIBRARY_PATH …" would take care of a lot, but is >>>> not really general. >>>> Is there a recommended way? >>>> regards, >>>> Michael >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > -- > -------------------------------------------------------------------------- > Dr David Singleton ANU Supercomputer Facility > HPC Systems Manager and NCI National Facility > david.single...@anu.edu.au Leonard Huxley Bldg (No. 56) > Phone: +61 2 6125 4389 Australian National University > Fax: +61 2 6125 8199 Canberra, ACT, 0200, Australia > -------------------------------------------------------------------------- > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
