Hi Min, Seems like the command ulimit was executed 16 times but after that (the "; ./wrf.exe") was ignored. Could you give the following a try:
bsub -e ERR -o OUT -n 16 "openmpi-mpirun \"/bin/sh -c 'ulimit -s unlimited ; ./wrf.exe ' \" " Somewhere, quoting goes wrong and I'm trying to figure out where.... Kind regards, Jeroen Kleijer On Thu, Dec 17, 2009 at 5:09 PM, Min Zhu <min....@res-ltd.com> wrote: > Hi, Jeroen, > > Here is the OUT file, ERR file is empty. > > -------------------------------------------------------------------------------------- > Sender: LSF System <lavaadmin@compute-10> > Subject: Job 662: <openmpi-mpirun /bin/sh -c 'ulimit -s unlimited; ./wrf.exe > ' > Done > > Job <openmpi-mpirun /bin/sh -c 'ulimit -s unlimited; ./wrf.exe ' > was > submitted from host <seamus> by user <mzh>. > Job was executed on host(s) <8*compute-10>, in queue <normal>, as user <mzh>. > <8*compute-11> > </home/mzh> was used as the home directory. > </home/mzh/wrf-intel/test/run1> was used as the working directory. > Started at Thu Dec 17 17:37:09 2009 > Results reported at Thu Dec 17 17:37:15 2009 > > Your job looked like: > > ------------------------------------------------------------ > # LSBATCH: User input > openmpi-mpirun /bin/sh -c 'ulimit -s unlimited; ./wrf.exe ' > ------------------------------------------------------------ > > Successfully completed. > > Resource usage summary: > > CPU time : 0.05 sec. > > The output (if any) follows: > > unlimited > unlimited > unlimited > unlimited > unlimited > unlimited > unlimited > unlimited > unlimited > unlimited > unlimited > unlimited > unlimited > unlimited > unlimited > unlimited > > > PS: > > Read file <ERR> for stderr output of this job. > > -------------------------------------------------- > > Thanks, > > Min Zhu > > > -----Original Message----- > From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On > Behalf Of Jeroen Kleijer > Sent: 17 December 2009 16:05 > To: Open MPI Users > Subject: Re: [OMPI users] About openmpi-mpirun > > Hi Min > > Did you get any type of error message in the ERR or OUT files? > I don't have mpirun installed in the environment at the moment but > giving the following: > > bsub -q interq -I "ssh <hostname> /bin/sh -c 'ulimit -s unlimited ; > /bin/hostname ' " > > seems to work for me, so I'm kind of curious what the error message is > you're seeing. > > Kind regards, > > Jeroen Kleijer > > On Thu, Dec 17, 2009 at 4:41 PM, Min Zhu <min....@res-ltd.com> wrote: >> Hi, Jeroen, >> >> Thanks for your reply. I tried the command bsub -e ERR -o OUT -n 16 >> "openmpi-mpirun /bin/sh -c 'ulimit -s unlimited; ./wrf.exe ' " and wrf.exe >> not executed. >> >> Cheers, >> >> Min Zhu >> >> -----Original Message----- >> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On >> Behalf Of Jeroen Kleijer >> Sent: 17 December 2009 15:34 >> To: Open MPI Users >> Subject: Re: [OMPI users] About openmpi-mpirun >> >> It's just that the "'s on the command line get parsed by LSF / bash >> (or whatever shell you use) >> >> If you wish to use it without the script you can give this a try: >> bsub -e ERR -o OUT -n 16 "openmpi-mpirun /bin/sh -c 'ulimit -s >> unlimited; ./wrf.exe ' " >> >> This causes to pass the whole string "openmpi-mpirun ...." to be >> passed as a single string / command to LSF. >> The second line between the single quotes is then passed as a single >> argument to /bin/sh which is run by openmpi-mpirun. >> >> Kind regards, >> >> Jeroen Kleijer >> >> On Thu, Dec 17, 2009 at 4:03 PM, Min Zhu <min....@res-ltd.com> wrote: >>> Hi, Jeff, >>> >>> Your script method works for me. Thank you very much, >>> >>> Cheers, >>> >>> Min Zhu >>> >>> >>> -----Original Message----- >>> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On >>> Behalf Of Jeff Squyres >>> Sent: 17 December 2009 14:56 >>> To: Open MPI Users >>> Subject: Re: [OMPI users] About openmpi-mpirun >>> >>> This might be something you need to talk to Platform about...? >>> >>> Another option would be to openmpi-mpirun a script that is just a few >>> lines long: >>> >>> #!/bin/sh >>> ulimit -s unlimited >>> ./wrf.exe >>> >>> >>> >>> On Dec 17, 2009, at 9:40 AM, Min Zhu wrote: >>> >>>> Hi, Jeff, >>>> >>>> Thanks. For bsub -e ERR -o OUT -n 16 openmpi-mpirun /bin/sh -c "ulimit >>>> -s unlimited; ./wrf.exe", I tried and wrf.exe doesn't executed. >>>> >>>> Here is the content of openmpi-mpirun file, so maybe something needs >>> to >>>> be changed? >>>> >>>> ---------------------------------------------- >>>> #!/bin/sh >>>> # >>>> # Copyright (c) 2007 Platform Computing >>>> # >>>> # This script is a wrapper for openmpi mpirun >>>> # it generates the machine file based on the hosts >>>> # given to it by Lava. >>>> # >>>> >>>> usage() { >>>> cat <<USEEOF >>>> USAGE: $0 >>>> This command is a wrapper for mpirun (openmpi). It can >>>> only be run within Lava using bsub e.g. >>>> bsub -n # "$0 -np # {my mpi command and args}" >>>> >>>> The wrapper will automatically generate the >>>> machinefile used by mpirun. >>>> >>>> NOTE: The list of hosts cannot exceed 4KBytes. >>>> USEEOF >>>> } >>>> >>>> if [ x"${LSB_JOBFILENAME}" = x -o x"${LSB_HOSTS}" = x ]; then >>>> usage >>>> exit -1 >>>> fi >>>> >>>> MYARGS=$* >>>> WORKDIR=`dirname ${LSB_JOBFILENAME}` >>>> MACHFILE=${WORKDIR}/mpi_machines >>>> ARGLIST=${WORKDIR}/mpi_args >>>> >>>> # Check if mpirun is in the PATH >>>> T=`which mpirun` >>>> if [ $? -ne 0 ]; then >>>> echo "Error: mpirun is not in your PATH." >>>> exit -2 >>>> fi >>>> >>>> echo "${MYARGS}" > ${ARGLIST} >>>> T=`grep -- -machinefile ${ARGLIST} |wc -l` >>>> if [ $T -gt 0 ]; then >>>> echo "Error: Do not provide the machinefile for mpirun." >>>> echo " It is generated automatically for you." >>>> exit -3 >>>> fi >>>> >>>> # Make the open-mpi machine file >>>> echo "${LSB_HOSTS}" > ${MACHFILE}.lst >>>> tr '\/ ' '\r\n' < ${MACHFILE}.lst > ${MACHFILE} >>>> >>>> MPIRUN=`which --skip-alias mpirun` >>>> ${MPIRUN} -x LD_LIBRARY_PATH -machinefile ${MACHFILE} ${MYARGS} >>>> >>>> exit $? >>>> >>>> ---------------------------------------------- >>>> >>>> >>>> Cheers, >>>> >>>> Min Zhu >>>> >>>> -----Original Message----- >>>> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] >>> On >>>> Behalf Of Jeff Squyres >>>> Sent: 17 December 2009 14:29 >>>> To: Open MPI Users >>>> Subject: Re: [OMPI users] About openmpi-mpirun >>>> >>>> On Dec 17, 2009, at 9:15 AM, Min Zhu wrote: >>>> >>>> > Thanks for your reply. Yes, your mpirun command works for me. But I >>>> need to use bsub job scheduler. I wonder why >>>> > bsub -e ERR -o OUT -n 16 openmpi-mpirun "/bin/sh -c ulimit -s >>>> unlimited; ./wrf.exe" doesn't work. >>>> >>>> Try with different quoting...? I don't know the details of the >>>> openmpi-mpirun script, but perhaps it's trying to exec the whole >>> quoted >>>> string as a single executable (which doesn't exist). Perhaps: >>>> >>>> bsub -e ERR -o OUT -n 16 openmpi-mpirun /bin/sh -c "ulimit -s >>> unlimited; >>>> ./wrf.exe" >>>> >>>> That's a (somewhat educated) guess... >>>> >>>> -- >>>> >>>> Jeff Squyres >>>> jsquy...@cisco.com >>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> CONFIDENTIALITY NOTICE: This e-mail, including any attachments, >>> contains information that may be confidential, and is protected by >>> copyright. It is directed to the intended recipient(s) only. If you >>> have received this e-mail in error please e-mail the sender by replying >>> to this message, and then delete the e-mail. Unauthorised disclosure, >>> publication, copying or use of this e-mail is prohibited. Any >>> communication of a personal nature in this e-mail is not made by or on >>> behalf of any RES group company. E-mails sent or received may be >>> monitored to ensure compliance with the law, regulation and/or our >>> policies. >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>> >>> >>> -- >>> >>> Jeff Squyres >>> jsquy...@cisco.com >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> CONFIDENTIALITY NOTICE: This e-mail, including any attachments, contains >>> information that may be confidential, and is protected by copyright. It is >>> directed to the intended recipient(s) only. If you have received this >>> e-mail in error please e-mail the sender by replying to this message, and >>> then delete the e-mail. Unauthorised disclosure, publication, copying or >>> use of this e-mail is prohibited. Any communication of a personal nature >>> in this e-mail is not made by or on behalf of any RES group company. >>> E-mails sent or received may be monitored to ensure compliance with the >>> law, regulation and/or our policies. >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> CONFIDENTIALITY NOTICE: This e-mail, including any attachments, contains >> information that may be confidential, and is protected by copyright. It is >> directed to the intended recipient(s) only. If you have received this >> e-mail in error please e-mail the sender by replying to this message, and >> then delete the e-mail. Unauthorised disclosure, publication, copying or use >> of this e-mail is prohibited. Any communication of a personal nature in >> this e-mail is not made by or on behalf of any RES group company. E-mails >> sent or received may be monitored to ensure compliance with the law, >> regulation and/or our policies. >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > CONFIDENTIALITY NOTICE: This e-mail, including any attachments, contains > information that may be confidential, and is protected by copyright. It is > directed to the intended recipient(s) only. If you have received this e-mail > in error please e-mail the sender by replying to this message, and then > delete the e-mail. Unauthorised disclosure, publication, copying or use of > this e-mail is prohibited. Any communication of a personal nature in this > e-mail is not made by or on behalf of any RES group company. E-mails sent or > received may be monitored to ensure compliance with the law, regulation > and/or our policies. > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
