[OMPI devel] (loose) SGE Integration fails, why?
Hi, I'm having some really strange error causing me some serious headaches. I want to integrate OpenMPI version 1.1.1 from the OFED package version 1.1 with SGE version 6.0. For mvapich all works, but for OpenMPI not ;(. Here is my jobfile and error message: #!/bin/csh -f #$ -N MPI_Job #$ -pe mpi 4 export PATH=$PATH:/usr/ofed/mpi/gcc/openmpi-1.1.1-1/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/ofed/mpi/gcc/openmpi-1.1.1.-1/lib64 /usr/ofed/mpi/gcc/openmpi-1.1.1-1/bin/mpirun -np $NSLOTS -hostfile $TMPDIR/machines /usr/ofed/mpi/gcc/openmpi-1.1.1-1/tests/IMB-2.3/IMB-MPI1 ERRORMESSAGE: [node04:25768] mca_mpool_openib_register: ibv_reg_mr(0x584000,102400) failed with error: Cannot allocate memory [node04:25768] mca_mpool_openib_register: ibv_reg_mr(0x584000,102400) failed with error: Cannot allocate memory [node04:25768] mca_mpool_openib_register: ibv_reg_mr(0x584000,528384) failed with error: Cannot allocate memory [node04:25768] mca_mpool_openib_register: ibv_reg_mr(0x584000,528384) failed with error: Cannot allocate memory [node04:25769] mca_mpool_openib_register: ibv_reg_mr(0x584000,102400) failed with error: Cannot allocate memory [node04:25769] mca_mpool_openib_register: ibv_reg_mr(0x584000,102400) failed with error: Cannot allocate memory [node04:25769] mca_mpool_openib_register: ibv_reg_mr(0x584000,528384) failed with error: Cannot allocate memory [node04:25769] mca_mpool_openib_register: ibv_reg_mr(0x584000,528384) failed with error: Cannot allocate memory [node04:25770] mca_mpool_openib_register: ibv_reg_mr(0x584000,102400) failed with error: Cannot allocate memory [node04:25770] mca_mpool_openib_register: ibv_reg_mr(0x584000,102400) failed with error: Cannot allocate memory [node04:25770] mca_mpool_openib_register: ibv_reg_mr(0x584000,528384) failed with error: Cannot allocate memory [node04:25770] mca_mpool_openib_register: ibv_reg_mr(0x584000,528384) failed with error: Cannot allocate memory [node04:25771] mca_mpool_openib_register: ibv_reg_mr(0x584000,102400) failed with error: Cannot allocate memory [node04:25771] mca_mpool_openib_register: ibv_reg_mr(0x584000,102400) failed with error: Cannot allocate memory [node04:25771] mca_mpool_openib_register: ibv_reg_mr(0x584000,528384) failed with error: Cannot allocate memory [node04:25771] mca_mpool_openib_register: ibv_reg_mr(0x584000,528384) failed with error: Cannot allocate memory [0,1,1][btl_openib.c:808:mca_btl_openib_create_cq_srq] error creating low priority cq for mthca0 errno says Cannot allocate memory -- It looks like MPI_INIT failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during MPI_INIT; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): PML add procs failed --> Returned "Error" (-1) instead of "Success" (0) -- *** An error occurred in MPI_Init *** before MPI was initialized *** MPI_ERRORS_ARE_FATAL (goodbye) MPI_Job.e111975 (END) If I run the OMPI job just with out SGE => everything works e.g. the following command: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/bin/mpirun -v -np 4 -H node04,node04,node04,node04 /usr/ofed/mpi/gcc/openmpi-1.1.1-1/tests/IMB-2.3/IMB-MPI1 If I do this with static machinefiles, it works too: $ cat /tmp/machines node04 node04 node04 node04 /usr/ofed/mpi/gcc/openmpi-1.1.1-1/bin/mpirun -v -np 4 -hostfile /tmp/machines /usr/ofed/mpi/gcc/openmpi-1.1.1-1/tests/IMB-2.3/IMB-MPI1 And if I run this in a jobscript it works even with a static machinefile (not shown below): #!/bin/csh -f #$ -N MPI_Job #$ -pe mpi 4 export PATH=$PATH:/usr/ofed/mpi/gcc/openmpi-1.1.1-1/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/ofed/mpi/gcc/openmpi-1.1.1.-1/lib64 /usr/ofed/mpi/gcc/openmpi-1.1.1-1/bin/mpirun -v -np 4 -H node04,node04,node04,node04 /usr/ofed/mpi/gcc/openmpi-1.1.1-1/tests/IMB-2.3/IMB-MPI1 There are correct ulimits for all nodes in the cluster e.g. node04: -sh-3.00$ ssh node04 ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited pending signals (-i) 1024 max locked memory (kbytes, -l) 8162952 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 139264 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited And the infiniband seems to have no troubles at all: -sh-3.00$ ibstat CA 'mthca0' CA type: MT25204 Number of ports: 1 Firmware version: 1.
Re: [OMPI devel] (loose) SGE Integration fails, why?
Hi, Jeff
many thanks for your reply..
> 1. You might want to update your version of Open MPI if possible; the
> v1.1.1 version is quite old. We have added many new bug fixes and
> features since v1.1.1 (including tight SGE integration). There is
> nothing special about the Open MPI that is included in the OFED
> distribution; you can download a new version from the Open MPI web
> site (the current stable version is v1.2.3), configure, compile, and
> install it with your current OFED installation. You should be able
> to configure Open MPI with:
Hmm, I've heard about conflicts with OMPI 1.2.x and OFED 1.1 (sorry no
refference here), and I've got no luck producing a working OMPI
installation ("mpirun --help" runs, and ./IMB-MPI compiles and runs too,
but "mpirun -np 2 node03,node14 IMB-MPI1" doesnt (segmentation
fault))... (beside that, I know that OFED 1.1 is quite old too) So I'm
tested it with OMPI 1.1.5 => same error.
> 2. I know little/nothing about SGE, but I'm assuming that you need to
> have SGE pass the proper memory lock limits to new processes. In an
> interactive login, you showed that the max limit is "8162952" -- you
> might just want to make it unlimited, unless you have a reason for
> limiting it. See http://www.open-mpi.org/faq/?
yes I allready read the faq, and even setting them to unlimited has
shown not be working. In the SGE one could specify the limits to
SGE-jobs by e.g. the qmon tool, (configuring queues > select queue >
modify > limits) But there is everything set to infinity. (Beside that,
the job is running with a static machinefile (is this an
"noninteractive" job?)) How could I test ulimits of interactive and
noninteractive jobs?
Thank you for your great help.
Re: [OMPI devel] (loose) SGE Integration fails, why?
Markus Daene schrieb: > Hi. > > I think it is not necessary to specify the hosts via the hostfile using SGE > and OpenMPI, even the $NSLOTS is not necessary , just run > mpirun executable this works very well. This produces the same error, but thanks for your suggestion. (For the sake of interest: how controls then ompi how many slots it may use?) > to your memory problem: > I had similar problems when I specified the h_vmem option to use in SGE. > Without SGE everything works, but starting with SGE gives such memory errors. > You can easily check this with 'qconf -sc'. If you have used this option, try > without it. The problem in my case was that OpenMPI allocates sometimes a lot > of memory and the job gets immediately killed by SGE, and one gets such error > messages, see my posting some days ago. I am not sure if this helps in your > case but it could be an explanation. Hmm it seems that I'm not using such an option (for my queue the h_vmem and s_vmem values are set to infinity). Here the output for the qconf -sc command. (Sorry for posting SGE related stuff on this mailing list): [~]# qconf -sc #name shortcut typerelop requestable consumable default urgency # archa RESTRING==YES NO NONE 0 calendarc RESTRING==YES NO NONE 0 cpu cpuDOUBLE >=YES NO 00 h_core h_core MEMORY <=YES NO 00 h_cpu h_cpu TIME<=YES NO 0:0:00 h_data h_data MEMORY <=YES NO 00 h_fsize h_fsizeMEMORY <=YES NO 00 h_rss h_rss MEMORY <=YES NO 00 h_rth_rt TIME<=YES NO 0:0:00 h_stack h_stackMEMORY <=YES NO 00 h_vmem h_vmem MEMORY <=YES NO 00 hostnameh HOST==YES NO NONE 0 load_avgla DOUBLE >=NO NO 00 load_long ll DOUBLE >=NO NO 00 load_medium lm DOUBLE >=NO NO 00 load_short ls DOUBLE >=NO NO 00 mem_freemf MEMORY <=YES NO 00 mem_total mt MEMORY <=YES NO 00 mem_usedmu MEMORY >=YES NO 00 min_cpu_intervalmciTIME<=NO NO 0:0:00 np_load_avg nlaDOUBLE >=NO NO 00 np_load_longnllDOUBLE >=NO NO 00 np_load_medium nlmDOUBLE >=NO NO 00 np_load_short nlsDOUBLE >=NO NO 00 num_procp INT ==YES NO 00 qname q RESTRING==YES NO NONE 0 rerun re BOOL==NO NO 00 s_core s_core MEMORY <=YES NO 00 s_cpu s_cpu TIME<=YES NO 0:0:00 s_data s_data MEMORY <=YES NO 00 s_fsize s_fsizeMEMORY <=YES NO 00 s_rss s_rss MEMORY <=YES NO 00 s_rts_rt TIME<=YES NO 0:0:00 s_stack s_stackMEMORY <=YES NO 00 s_vmem s_vmem MEMORY <=YES NO 00 seq_no seqINT ==NO NO 00 slots s INT <=YES YES 11000 swap_free sf MEMORY <=YES NO 00 swap_rate sr MEMORY >=YES NO 00 swap_rsvd srsv MEMORY >=YES NO 00 swap_total st MEMORY <=YES NO 00 swap_used su MEMORY >=YES NO 00 tmpdir tmpRESTRING==NO NO NONE 0 virtual_freevf MEMORY <=YES NO 00 virtual_total vt MEMORY <=YES NO 00 virtual_usedvu MEMORY >=YES NO 00 # >#< starts a comment but comments are not saved across edits thanks for your help.
Re: [OMPI devel] (loose) SGE Integration fails, why?
Markus Daene wrote: >>> to your memory problem: >>> I had similar problems when I specified the h_vmem option to use in SGE. >>> Without SGE everything works, but starting with SGE gives such memory >>> errors. You can easily check this with 'qconf -sc'. If you have used this >>> option, try without it. The problem in my case was that OpenMPI allocates >>> sometimes a lot of memory and the job gets immediately killed by SGE, and >>> one gets such error messages, see my posting some days ago. I am not sure >>> if this helps in your case but it could be an explanation. > > I am sorry to discuss SGE stuff here as well, but there was this question and > one should make clear that this is not just related to OMPI. > > I think your output shows exactely the problem: you have set h_vmem as > requestable and the default value to 0, the job has no memory at all. OMPI (thought that zero means infinity) > somehow knows that is has just this memory granted by SGE, so it cannot > allocate any memory in this case. Of course you get the errors. > You should either set h_vmem to not requestable, or set a proper default > value. e.g. 2.0G, or specify the memory consumption in your job script like > #$ -l h_vmem=2000M > it is not important that your queue has set h_vmem to infinity, this gives > you > just the maximum which you can request. If I use the h_vmem option I get a slight different error, but if I mark h_vmem as not requestable => same error. Below is the slight different error message: [node17:02861] mca: base: component_find: unable to open: libsysfs.so.1: failed to map segment from shared object: Cannot allocate memory (ignored) [node17:02861] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_pml_ob1.so: failed to map segment from shared o bject: Cannot allocate memory (ignored) [node17:02861] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_coll_basic.so: failed to map segment from share d object: Cannot allocate memory (ignored) [node17:02861] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_coll_hierarch.so: failed to map segment from sh ared object: Cannot allocate memory (ignored) [node17:02861] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_coll_self.so: failed to map segment from shared object: Cannot allocate memory (ignored) [node17:02861] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_coll_sm.so: failed to map segment from shared o bject: Cannot allocate memory (ignored) [node17:02861] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_coll_tuned.so: failed to map segment from share d object: Cannot allocate memory (ignored) [node17:02861] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_osc_pt2pt.so: failed to map segment from shared object: Cannot allocate memory (ignored) [node17:02862] mca: base: component_find: unable to open: libsysfs.so.1: failed to map segment from shared object: Cannot allocate memory (ignored) [node17:02862] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_pml_ob1.so: failed to map segment from shared o bject: Cannot allocate memory (ignored) [node17:02862] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_coll_basic.so: failed to map segment from share d object: Cannot allocate memory (ignored) [node17:02862] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_coll_hierarch.so: failed to map segment from sh ared object: Cannot allocate memory (ignored) [node17:02862] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_coll_self.so: failed to map segment from shared object: Cannot allocate memory (ignored) [node17:02862] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_coll_sm.so: failed to map segment from shared o bject: Cannot allocate memory (ignored) [node17:02862] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_coll_tuned.so: failed to map segment from share d object: Cannot allocate memory (ignored) [node17:02862] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_osc_pt2pt.so: failed to map segment from shared object: Cannot allocate memory (ignored) [node17:02863] mca: base: component_find: unable to open: libsysfs.so.1: failed to map segment from shared object: Cannot allocate memory (ignored) [node17:02863] mca: base: component_find: unable to open: /usr/ofed/mpi/gcc/openmpi-1.1.1-1/lib64/openmpi/mca_pml_ob1.so: failed to map segment from shared o bject: Cannot allocate memory (ignored) [node17:02863] mca: base: componen
Re: [OMPI devel] (loose) SGE Integration fails, why?
Hi Pak, > Jeff Squyres wrote: 2. I know little/nothing about SGE, but I'm assuming that you need to have SGE pass the proper memory lock limits to new processes. In an interactive login, you showed that the max limit is "8162952" -- you might just want to make it unlimited, unless you have a reason for limiting it. See http://www.open-mpi.org/faq/? >>> yes I allready read the faq, and even setting them to unlimited has >>> shown not be working. In the SGE one could specify the limits to >>> SGE-jobs by e.g. the qmon tool, (configuring queues > select queue > >>> modify > limits) But there is everything set to infinity. (Beside >>> that, >>> the job is running with a static machinefile (is this an >>> "noninteractive" job?)) How could I test ulimits of interactive and >>> noninteractive jobs? >> Launch an SGE job that calls the shell command "limit" (if you run C- >> shell variants) or "ulimit -l" (if you run Bourne shell variants). >> Ensure that the output is "unlimited". >> >> What are the limits of the user that launches the SGE daemons? I.e., >> did the SGE daemons get started with proper "unlimited" limits? If >> not, that could hamper SGE's ability to set the limits that you told >> it to via qmon (remember my disclaimer: I know nothing about SGE, so >> this is speculation). >> > > I am assuming you have tried without using SGE (like via ssh or others) > to launch your job and that works correctly? If yes then you should > compare the outputs of limit as Jeff suggested to see if they are any > difference between the two (with and without using SGE). Yes, without SGE all works, with SGE it does work too if I use a static machinefile (see initial post), or -H h1,...,hn does work too! Just with the SGE's generate $TMPDIR/machines file (which in turn is valid! I checked this), the job doesn't run. And the ulimits are (in every three possibilities every time) unlimited: pos1: pdsh -R shh -w node[XX-YY] ulimit -a => unlimited (loose coupled) pos2: qsub jobscribt, where jobscript just calls the command as in pos1 (thight coupled?) pos3: qsub jobscribt, where jobscript calls another script (containing the same command as in pos1) and additionally passing $TMPDIR/machines as argument to it. Thanks for your help.
Re: [OMPI devel] (loose) SGE Integration fails, why?
Jeff Squyres schrieb:
>> Hmm, I've heard about conflicts with OMPI 1.2.x and OFED 1.1 (sorry no
>> refference here),
>
> I'm unaware of any problems with OMPI 1.2.x and OFED 1.1. I run OFED
> 1.1 on my cluster at Cisco and have many different versions of OMPI
> installed (1.2, trunk, etc.).
Yes you are right, I read wrong (in the OMPI 1.2 changelog (README) OFED
1.0 isn't considered to work with OMPI 1.2. Sorry..).
>> and I've got no luck producing a working OMPI
>> installation ("mpirun --help" runs, and ./IMB-MPI compiles and runs
>> too,
>> but "mpirun -np 2 node03,node14 IMB-MPI1" doesnt (segmentation
>> fault))...
>
> Can you send more information on this? See http://www.open-mpi.org/
> community/help/
-sh-3.00$ ompi/bin/mpirun -d -np 2 -H node03,node06 hostname
[headnode:23178] connect_uni: connection not allowed
[headnode:23178] connect_uni: connection not allowed
[headnode:23178] connect_uni: connection not allowed
[headnode:23178] connect_uni: connection not allowed
[headnode:23178] connect_uni: connection not allowed
[headnode:23178] connect_uni: connection not allowed
[headnode:23178] connect_uni: connection not allowed
[headnode:23178] connect_uni: connection not allowed
[headnode:23178] connect_uni: connection not allowed
[headnode:23178] connect_uni: connection not allowed
[headnode:23178] [0,0,0] setting up session dir with
[headnode:23178]universe default-universe-23178
[headnode:23178]user me
[headnode:23178]host headnode
[headnode:23178]jobid 0
[headnode:23178]procid 0
[headnode:23178] procdir:
/tmp/openmpi-sessions-me@headnode_0/default-universe-23178/0/0
[headnode:23178] jobdir:
/tmp/openmpi-sessions-me@headnode_0/default-universe-23178/0
[headnode:23178] unidir:
/tmp/openmpi-sessions-me@headnode_0/default-universe-23178
[headnode:23178] top: openmpi-sessions-me@headnode_0
[headnode:23178] tmp: /tmp
[headnode:23178] [0,0,0] contact_file
/tmp/openmpi-sessions-me@headnode_0/default-universe-23178/universe-setup.txt
[headnode:23178] [0,0,0] wrote setup file
[headnode:23178] *** Process received signal ***
[headnode:23178] Signal: Segmentation fault (11)
[headnode:23178] Signal code: Address not mapped (1)
[headnode:23178] Failing at address: 0x1
[headnode:23178] [ 0] /lib64/tls/libpthread.so.0 [0x39ed80c430]
[headnode:23178] [ 1] /lib64/tls/libc.so.6(strcmp+0) [0x39ecf6ff00]
[headnode:23178] [ 2]
/home/me/ompi/lib/openmpi/mca_pls_rsh.so(orte_pls_rsh_launch+0x24f)
[0x2a9723cc7f]
[headnode:23178] [ 3] /home/me/ompi/lib/openmpi/mca_rmgr_urm.so
[0x2a9764fa90]
[headnode:23178] [ 4] /home/me/ompi/bin/mpirun(orterun+0x35b) [0x402ca3]
[headnode:23178] [ 5] /home/me/ompi/bin/mpirun(main+0x1b) [0x402943]
[headnode:23178] [ 6] /lib64/tls/libc.so.6(__libc_start_main+0xdb)
[0x39ecf1c3fb]
[headnode:23178] [ 7] /home/me/ompi/bin/mpirun [0x40289a]
[headnode:23178] *** End of error message ***
Segmentation fault
>> yes I allready read the faq, and even setting them to unlimited has
>> shown not be working. In the SGE one could specify the limits to
>> SGE-jobs by e.g. the qmon tool, (configuring queues > select queue >
>> modify > limits) But there is everything set to infinity. (Beside
>> that,
>> the job is running with a static machinefile (is this an
>> "noninteractive" job?)) How could I test ulimits of interactive and
>> noninteractive jobs?
>
> Launch an SGE job that calls the shell command "limit" (if you run C-
> shell variants) or "ulimit -l" (if you run Bourne shell variants).
> Ensure that the output is "unlimited".
I've done that allready, but how to distinguish between tight coupled
job ulimits and loose coupled job ulimits? I tested to pass
$TMPDIR/machines to a shell script which in turn delivers a "ulimit -a",
*assuming* this is considered as a tight coupled job, but each node
returned unlimited.. and without this $TMPDIR/machines too. Even the
headnode is set to unlimited.
> What are the limits of the user that launches the SGE daemons? I.e.,
> did the SGE daemons get started with proper "unlimited" limits? If
> not, that could hamper SGE's ability to set the limits that you told
The limits in /etc/security/limits.conf apply to all users (using a
'*'), hence the SGE processes and deamons shouldn't have any limits.
> it to via qmon (remember my disclaimer: I know nothing about SGE, so
> this is speculation).
But thanks anyway => I will post this issue to an SGE mailing list soon.
The config.log and the `ompi_info --all` is attached. Thanks again to
all of you.
logs.tbz
Description: application/bzip-compressed-tar
Re: [OMPI devel] (loose) SGE Integration fails, why?
Sorry for late reply, but I havent had access to the machine at the weekend. > I don't really know what this means. People have explained "loose" > vs. "tight" integration to me before, but since I'm not an SGE user, > the definitions always fall away. I *assume* loose coupled jobs, are just jobs, where the SGE find some nodes to process them and from then on, it doesn't care about anything in conjunction to the jobs. In contrast to tight coupled jobs, where the SGE take care for sub process which could spwan from the job and terminate them too in case of a failure, or take care of specified resources. > Based on your prior e-mail, it looks like you are always invoking > "ulimit" via "pdsh", even under SGE jobs. This is incorrect. why? > Can't you just submit an SGE job script that runs "ulimit"? #!/bin/csh -f #$ -N MPI_Job #$ -pe mpi 4 hostname && ulimit -a ATM I'm quite confused: cause I want to use the c-shell, but ulimit is just for bash. The c-shell uses limit... hmm.. and SGE uses obviously bash, instead of my request for csh in the first line. But if I just use #!/bin/bash I get the same limits: -sh-3.00$ cat MPI_Job.o112116 node02 core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited pending signals (-i) 1024 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 stack size (kbytes, -s) unlimited cpu time (seconds, -t) unlimited max user processes (-u) 139264 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited oops => 32 kbytes... So this isn't OMPI's fault. >>> What are the limits of the user that launches the SGE daemons? I.e., >>> did the SGE daemons get started with proper "unlimited" limits? If >>> not, that could hamper SGE's ability to set the limits that you told >> The limits in /etc/security/limits.conf apply to all users (using a >> '*'), hence the SGE processes and deamons shouldn't have any limits. > > Not really. limits.conf is not universally applied; it's a PAM > entity. So for daemons that start via /etc/init.d scripts (or > whatever the equivalent is on your system), PAM limits are not > necessarily applied. For example, I had to manually insert a "ulimit > -Hl unlimited" in the startup script for my SLURM daemons. Hmm, ATM there are some important jobs in the queue (started some MONTHS ago) so I cannot restart the daemon. Is there any other way than restart (with proper limits) for ensuring the limits of a process? thanks for your great help :)
Re: [OMPI devel] (loose) SGE Integration fails, why?
Pak Lui schrieb: > sad...@gmx.net wrote: >> Sorry for late reply, but I havent had access to the machine at the weekend. >> >>> I don't really know what this means. People have explained "loose" >>> vs. "tight" integration to me before, but since I'm not an SGE user, >>> the definitions always fall away. >> I *assume* loose coupled jobs, are just jobs, where the SGE find some >> nodes to process them and from then on, it doesn't care about anything >> in conjunction to the jobs. In contrast to tight coupled jobs, where the >> SGE take care for sub process which could spwan from the job and >> terminate them too in case of a failure, or take care of specified >> resources. >> >>> Based on your prior e-mail, it looks like you are always invoking >>> "ulimit" via "pdsh", even under SGE jobs. This is incorrect. >> why? >> >>> Can't you just submit an SGE job script that runs "ulimit"? >> #!/bin/csh -f >> #$ -N MPI_Job >> #$ -pe mpi 4 >> hostname && ulimit -a >> >> ATM I'm quite confused: cause I want to use the c-shell, but ulimit is >> just for bash. The c-shell uses limit... hmm.. and SGE uses obviously >> bash, instead of my request for csh in the first line. But if I just use >> #!/bin/bash I get the same limits: >> >> -sh-3.00$ cat MPI_Job.o112116 >> node02 >> core file size (blocks, -c) unlimited >> data seg size (kbytes, -d) unlimited >> file size (blocks, -f) unlimited >> pending signals (-i) 1024 >> max locked memory (kbytes, -l) 32 >> max memory size (kbytes, -m) unlimited >> open files (-n) 1024 >> pipe size(512 bytes, -p) 8 >> POSIX message queues (bytes, -q) 819200 >> stack size (kbytes, -s) unlimited >> cpu time (seconds, -t) unlimited >> max user processes (-u) 139264 >> virtual memory (kbytes, -v) unlimited >> file locks (-x) unlimited >> >> oops => 32 kbytes... So this isn't OMPI's fault. > > this looks like sge_execd isn't able to source the correct system > defaults from the limit.conf file after you applied the change. Maybe > you will need to restart the daemon? Yes I posted the same question to the sun grid engine mailing list, and as Jeff initially supposed it was the inproper limits for the daemons (sgeexec). So I've to edit each node's init script (/etc/init.d/sgeexecd), and put "ulimit -l unlimited" before starting sge_execd. Then kill all sgeexecd's (running jobs won't be affected if you use "qconf -ke all") then restart every node's sgeexecd. After that every thing with SGE and OMPI 1.1.1 was fine. But for the whole question just read the small thread at: http://gridengine.sunsource.net/servlets/ReadMsg?list=users&msgNo=20390 At this point big thanks to Jeff, and all other which helped me! Are there any suggestions to the compilation error? many many thousand thanks for the great help here in the forum!
Re: [OMPI devel] (loose) SGE Integration fails, why?
> Are you referring to this SEGV error here? I am assuming this is OMPI > 1.1.1 so you are using rsh PLS to launch your executables (using loose > integration). oops, I wanted to compile ompi 1.2.3 against OFED 1.1 and these are the errors. This problem has nothing to do with the SGE anymore (Jeff suggested me to migrate to a "slightly" newer version, so I tried and failed with these errors) Should I start a whole new thread on this, since the SGE question is solved? > >-sh-3.00$ ompi/bin/mpirun -d -np 2 -H node03,node06 hostname > > [headnode:23178] connect_uni: connection not allowed > > [headnode:23178] connect_uni: connection not allowed > > [headnode:23178] connect_uni: connection not allowed > > [headnode:23178] connect_uni: connection not allowed > > [headnode:23178] connect_uni: connection not allowed > > [headnode:23178] connect_uni: connection not allowed > > [headnode:23178] connect_uni: connection not allowed > > [headnode:23178] connect_uni: connection not allowed > > [headnode:23178] connect_uni: connection not allowed > > [headnode:23178] connect_uni: connection not allowed > > [headnode:23178] [0,0,0] setting up session dir with > > [headnode:23178]universe default-universe-23178 > > [headnode:23178]user me > > [headnode:23178]host headnode > > [headnode:23178]jobid 0 > > [headnode:23178]procid 0 > > [headnode:23178] procdir: > > /tmp/openmpi-sessions-me@headnode_0/default-universe-23178/0/0 > > [headnode:23178] jobdir: > > /tmp/openmpi-sessions-me@headnode_0/default-universe-23178/0 > > [headnode:23178] unidir: > > /tmp/openmpi-sessions-me@headnode_0/default-universe-23178 > > [headnode:23178] top: openmpi-sessions-me@headnode_0 > > [headnode:23178] tmp: /tmp > > [headnode:23178] [0,0,0] contact_file > > /tmp/openmpi-sessions-me@headnode_0/default-universe-23178/universe- > > setup.txt > > [headnode:23178] [0,0,0] wrote setup file > > [headnode:23178] *** Process received signal *** > > [headnode:23178] Signal: Segmentation fault (11) > > [headnode:23178] Signal code: Address not mapped (1) > > [headnode:23178] Failing at address: 0x1 > > [headnode:23178] [ 0] /lib64/tls/libpthread.so.0 [0x39ed80c430] > > [headnode:23178] [ 1] /lib64/tls/libc.so.6(strcmp+0) [0x39ecf6ff00] > > [headnode:23178] [ 2] > > /home/me/ompi/lib/openmpi/mca_pls_rsh.so(orte_pls_rsh_launch+0x24f) > > [0x2a9723cc7f] > > [headnode:23178] [ 3] /home/me/ompi/lib/openmpi/mca_rmgr_urm.so > > [0x2a9764fa90] > > [headnode:23178] [ 4] /home/me/ompi/bin/mpirun(orterun+0x35b) > > [0x402ca3] > > [headnode:23178] [ 5] /home/me/ompi/bin/mpirun(main+0x1b) [0x402943] > > [headnode:23178] [ 6] /lib64/tls/libc.so.6(__libc_start_main+0xdb) > > [0x39ecf1c3fb] > > [headnode:23178] [ 7] /home/me/ompi/bin/mpirun [0x40289a] > > [headnode:23178] *** End of error message *** > > Segmentation fault > > So is it true that SEGV only occurred under the SGE environment and not > a normal environment? If it is then I am baffled because starting rsh > pls under the SGE environment in 1.1.1 should be no different than > starting rsh pls without SGE. nope the config.log and "ompi_info --all" output are attached some posts before. Sorry for this topic confusion. thank you.
