Re: [OMPI users] ompi + bash + GE + modules
Hi Mark, Have you tried adding -l to the #! line? #!/bin/bash -l On Wed, Jan 11, 2012 at 10:42 AM, Mark Suhovecky wrote: > #!/bin/bash > #$ > > module load ompi > > mpiexec > > when the mpiexec is run, we'll see the following errors > > > bash: module: line 1: syntax error: unexpected end of file > bash: error importing function definition for `module' > -- Edmund Sumbar Research Computing Support University of Alberta +1 780 492 9360
Re: [OMPI users] ompi + bash + GE + modules
Edmund- Yeah, I've tried that. No difference. Our template .bash_profile sources the user's .bashrc, so non-interactive bash shells in our setup are sourcing .bashrc. The modules environment is defined, and works- only jonbs that run across multiple machines see this error. Mark Suhovecky HPC System Administrator Center for Research Computing University of Notre Dame suhove...@nd.edu From: users-boun...@open-mpi.org [users-boun...@open-mpi.org] On Behalf Of Edmund Sumbar [esum...@ualberta.ca] Sent: Wednesday, January 11, 2012 12:52 PM To: Open MPI Users Subject: Re: [OMPI users] ompi + bash + GE + modules Hi Mark, Have you tried adding -l to the #! line? #!/bin/bash -l On Wed, Jan 11, 2012 at 10:42 AM, Mark Suhovecky mailto:suhove...@nd.edu>> wrote: #!/bin/bash #$ module load ompi mpiexec when the mpiexec is run, we'll see the following errors bash: module: line 1: syntax error: unexpected end of file bash: error importing function definition for `module' -- Edmund Sumbar Research Computing Support University of Alberta +1 780 492 9360
Re: [OMPI users] ompi + bash + GE + modules
Hi Mark
I wonder if you need to initialize the module command environment inside your
SGE
bash submission script:
$MODULESHOME/init/
where is bash in this case. See 'man module' for more details.
This would be before you actually invoke the module command:
module load openmpi
I am guessing your users' default shell is csh, and they may perhaps have a
'csh'
module environment initialized by their .cshrc, but the job submission script
is in bash.
Anyway, we use Torque, not SGE, so this is just a guess.
I hope it helps,
Gus Correa
On Jan 11, 2012, at 12:42 PM, Mark Suhovecky wrote:
>
> Hi-
>
> We run OpenMPI 1.4.3 on RHEL5 in a cluster environment.
> We use Univa Grid Engine 8.0.1 (an SGE spinoff) for job submission.
> We've just recently begun supporting the bash shell for submitted jobs,
> and are seeing a problem with submitted MPI jobs.
>
> Our software environment is manged with Modules package (version 3.2.8),
> so a typical job submission looks something like this
>
> #!/bin/bash
> #$
>
> module load ompi
>
> mpiexec
>
> when the mpiexec is run, we'll see the following errors
>
>
> bash: module: line 1: syntax error: unexpected end of file
> bash: error importing function definition for `module'
>
> The module int file contains this function, which is what I'm assuming all
> the fuss is about:
>
> module() { eval `/opt/crc/Modules/$MODULE_VERSION/bin/modulecmd bash $*`; }
> export -f module
>
> There will be multiple instances of the error generated- for example, if I'm
> running a 48 core mpi-12 job spread across 4 machines,
> I'll see these errors printed 3 times. I don't see these errors
> on single-machine submitted jobs.
>
> I've found posts for this error on bash, modules, and SGE lists, and have
> tried a number of suggested workarounds that all involve changing how I
> source modules (in /etc/profile.d, .bash_profile, via BASH_ENV), but
> none have gotten rid of this error.
>
> Since we only see this problem with MPI, I figured it couldn't hurt to post
> here and see if any of you have had this symptom, and what your solution was.
>
> I should mention that running a submitted MPI job under csh works just fine.
>
> Thanks for any help,
>
> Mark
>
> Mark Suhovecky
> HPC System Administrator
> Center for Research Computing
> University of Notre Dame
> suhovecky at nd.edu
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] ompi + bash + GE + modules
Hi,
Am 11.01.2012 um 19:12 schrieb Mark Suhovecky:
> Edmund-
>
> Yeah, I've tried that. No difference. Our template .bash_profile sources
> the user's .bashrc, so non-interactive bash shells in our setup are sourcing
> .bashrc.
SGE 6.2u5 can't handle multi line environment variables or functions, it was
fixed in 6.2u6 which isn't free. Do you use -V while submitting the job? Just
ignore the error or look into Son of Gridengine which fixed it too.
If you can avoid -V, then it could be defined in any of the .profile or alike
if you use -l as suggested. You could even define a started_method in SGE to
define it for all users by default and avoid to use -V:
#!/bin/sh
module() { ...command...here... }
export -f module
exec "${@}"
-- Reuti
> The modules environment is defined, and works- only jonbs that run across
> multiple machines see this error.
>
> Mark Suhovecky
> HPC System Administrator
> Center for Research Computing
> University of Notre Dame
> suhove...@nd.edu
>
> From: users-boun...@open-mpi.org [users-boun...@open-mpi.org] On Behalf Of
> Edmund Sumbar [esum...@ualberta.ca]
> Sent: Wednesday, January 11, 2012 12:52 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] ompi + bash + GE + modules
>
> Hi Mark,
>
> Have you tried adding -l to the #! line?
>
> #!/bin/bash -l
>
> On Wed, Jan 11, 2012 at 10:42 AM, Mark Suhovecky
> mailto:suhove...@nd.edu>> wrote:
> #!/bin/bash
> #$
>
> module load ompi
>
> mpiexec
>
> when the mpiexec is run, we'll see the following errors
>
>
> bash: module: line 1: syntax error: unexpected end of file
> bash: error importing function definition for `module'
>
>
>
> --
> Edmund Sumbar
> Research Computing Support
> University of Alberta
> +1 780 492 9360
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] ompi + bash + GE + modules
Surely this should be on the gridengine list -- and it's in recent
archives -- but there's some ob-openmpi below. Can Notre Dame not get
the support they've paid Univa for?
Reuti writes:
> SGE 6.2u5 can't handle multi line environment variables or functions,
> it was fixed in 6.2u6 which isn't free.
[It's not listed for 6.2u6.] For what it's worth, my fix for Sun's fix
is https://arc.liv.ac.uk/trac/SGE/changeset/3556/sge.
> Do you use -V while submitting the job? Just ignore the error or look
> into Son of Gridengine which fixed it too.
Of course
you can always avoid the issue by not using `export -f', which isn't in
the modules version we have. I default -V in sge_request and load
the open-mpi module in the job submission session. I don't
fin whatever problems it causes, and it works for binaries like
qsub -b y ... mpirun ...
However, the folkloristic examples here typically load the module stuff
in the job script.
> If you can avoid -V, then it could be defined in any of the .profile
> or alike if you use -l as suggested. You could even define a
> started_method in SGE to define it for all users by default and avoid
> to use -V:
>
> #!/bin/sh
> module() { ...command...here... }
> export -f module
> exec "${@}"
That won't work for example if someone is tasteless enough to submit csh.
Re: [OMPI users] ompi + bash + GE + modules
Dave-
I'm working with Univa support as well.
I started out debugging this with pretty poor grasp of where in the software
flow the problem
might be. Like most sysadmins, I belong to many community lists, and find them
to be of tremendous
help in running problems down. They certainly have been in this case- I've
posted to the modules-interest
sourcefourge group as well.
I choose to use all the resources open to me, including community user forums
and paid support, Using
a commercial product's support should not preclude one from using other tools
as well.
Mark
Mark Suhovecky
HPC System Administrator
Center for Research Computing
University of Notre Dame
suhove...@nd.edu
From: users-boun...@open-mpi.org [users-boun...@open-mpi.org] On Behalf Of Dave
Love [d.l...@liverpool.ac.uk]
Sent: Thursday, January 12, 2012 8:40 AM
To: us...@open-mpi.org
Subject: Re: [OMPI users] ompi + bash + GE + modules
Surely this should be on the gridengine list -- and it's in recent
archives -- but there's some ob-openmpi below. Can Notre Dame not get
the support they've paid Univa for?
Reuti writes:
> SGE 6.2u5 can't handle multi line environment variables or functions,
> it was fixed in 6.2u6 which isn't free.
[It's not listed for 6.2u6.] For what it's worth, my fix for Sun's fix
is https://arc.liv.ac.uk/trac/SGE/changeset/3556/sge.
> Do you use -V while submitting the job? Just ignore the error or look
> into Son of Gridengine which fixed it too.
Of course
you can always avoid the issue by not using `export -f', which isn't in
the modules version we have. I default -V in sge_request and load
the open-mpi module in the job submission session. I don't
fin whatever problems it causes, and it works for binaries like
qsub -b y ... mpirun ...
However, the folkloristic examples here typically load the module stuff
in the job script.
> If you can avoid -V, then it could be defined in any of the .profile
> or alike if you use -l as suggested. You could even define a
> started_method in SGE to define it for all users by default and avoid
> to use -V:
>
> #!/bin/sh
> module() { ...command...here... }
> export -f module
> exec "${@}"
That won't work for example if someone is tasteless enough to submit csh.
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] ompi + bash + GE + modules
On 01/12/2012 08:40 AM, Dave Love wrote:
> Surely this should be on the gridengine list -- and it's in recent
> archives -- but there's some ob-openmpi below. Can Notre Dame not get
> the support they've paid Univa for?
This is, in fact, in the recent gridengine archives. I brought up this
problem myself within the past couple of months ago.
> Reuti writes:
>
>> SGE 6.2u5 can't handle multi line environment variables or functions,
>> it was fixed in 6.2u6 which isn't free.
> [It's not listed for 6.2u6.] For what it's worth, my fix for Sun's fix
> is https://arc.liv.ac.uk/trac/SGE/changeset/3556/sge.
>
>> Do you use -V while submitting the job? Just ignore the error or look
>> into Son of Gridengine which fixed it too.
> Of course
> you can always avoid the issue by not using `export -f', which isn't in
> the modules version we have. I default -V in sge_request and load
> the open-mpi module in the job submission session. I don't
> fin whatever problems it causes, and it works for binaries like
> qsub -b y ... mpirun ...
> However, the folkloristic examples here typically load the module stuff
> in the job script.
>
>> If you can avoid -V, then it could be defined in any of the .profile
>> or alike if you use -l as suggested. You could even define a
>> started_method in SGE to define it for all users by default and avoid
>> to use -V:
>>
>> #!/bin/sh
>> module() { ...command...here... }
>> export -f module
>> exec "${@}"
> That won't work for example if someone is tasteless enough to submit csh.
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
