Hi, > Am 15.08.2016 um 14:30 schrieb Ulrich Hiller <[email protected]>: > > Hello, > >> The other issue seems to be, that in fact your job is using only one > machine, which means that it is essentially ignoring any granted slot > allocation. While the job is running, can you please execute on the > master node of the parallel job: >> >> $ ps -e f >> >> (f w/o -) and post the relevant lines belonging to either sge_execd or > just running as kids of the init process, in case they jumped out of the > process tree. Maybe a good start would be to execute something like > `mpiexec sleep 300` in the jobscript. >> > > i invoked > qsub -pe orte 160 -j yes -cwd -S /bin/bash <<< "mpiexec -n 160 sleep 300" > > the only line ('ps -e f') on the master node was: > 55722 ? Sl 3:42 /opt/sge/bin/lx-amd64/sge_qmaster
I meant the master node of the parallel job. According to the output below it was exec-node01 in your test. > No other sge lines, no child processes from it, and no other init > processes leading to sge While at the same time the sleep processes were > running on the nodes (Checked with ps command on the nodes). > > The qstat command gave : > 264 0.60500 STDIN ulrich r 08/15/2016 11:33:02 > all.q@exec-node01 MASTER Here you should see some processes executing a `qrsh -inherit ...` Does is gives a better output? > all.q@exec-node01 SLAVE > > all.q@exec-node01 SLAVE > > all.q@exec-node01 SLAVE > [ ...] > > 264 0.60500 STDIN ulrich r 08/15/2016 11:33:02 > all.q@exec-node03 SLAVE > > all.q@exec-node03 SLAVE > > all.q@exec-node03 SLAVE > [ ... ] > 264 0.60500 STDIN ulrich r 08/15/2016 11:33:02 > all.q@exec-node05 SLAVE > > all.q@exec-node05 SLAVE > [ ...] The output of SGE shows only what was granted. Whether parallel complies to it, is often a necessary setting to look into and adjust in case it's not working as intended. Therefore the checks with `ps -e f`. -- Reuti > Because there was only the master deamon running on the master node, and > you were tlaking about child processes: Was this now normal behaviour my > cluster showed or is there something wrong? > > Kind reagrds, ulrich > > > > On 08/12/2016 07:11 PM, Reuti wrote: >> Hi, >> >>> Am 12.08.2016 um 18:48 schrieb Ulrich Hiller <[email protected]>: >>> >>> Hello, >>> >>> i have a strange effect, where i am not sure whether it is "only" a >>> misconfiguration or a bug. >>> >>> First: I run son of gridengine 8.1.9-1.el6.x86_64 (i installed the rhel >>> rpm on an opensuse 13.1 machine. This should not matter in this case, >>> and it is reported to be able to run on opensuse). >>> >>> mpirun and mpiexec are from openmpi-1.10.3 (no other mpi was installed, >>> neither on master, nor on slaves). The installation was made with: >>> ./configure --prefix=`pwd`/build --disable-dlopen --disable-mca-dso >>> --with-orte --with-sge --with-x --enable-mpi-thread-multiple >>> --enable-orterun-prefix-by-default --enable-mpirun-prefix-by-default >>> --enable-orte-static-ports --enable-mpi-cxx --enable-mpi-cxx-seek >>> --enable-oshmem --enable-java --enable-mpi-java >>> make >>> make install >>> >>> I attached the outputs of 'qconf -ap all.q' , 'qconf -sconf' and 'qconf >>> -sp orte' as textfiles. >>> >>> Now my problem: >>> I asked for 20 cores and if i run qstat -u '*' it shows that this job >>> is being run in slave07 using 20 cores but is not true! if i run qstat >>> -f -u '*' i see that this job is only using 3 cores in salve07 and >>> there are 17 cores in other nodes allocated to this job which are in fact >>> unused! >> >> qstat will list only the master node of the parallel job and the number of >> overall slots. The granted allocation you can check with: >> >> $ qstat -g t -u '*' >> >> The other issue seems to be, that in fact your job is using only one >> machine, which means that it is essentially ignoring any granted slot >> allocation. While the job is running, can you please execute on the master >> node of the parallel job: >> >> $ ps -e f >> >> (f w/o -) and post the relevant lines belonging to either sge_execd or just >> running as kids of the init process, in case they jumped out of the process >> tree. Maybe a good start would be to execute something like `mpiexec sleep >> 300` in the jobscript. >> >> Next step could be a `mpihello.c` where you put an almost endless loop >> inside and switch off all optimizations during compilations to check whether >> these slave processes are distributed in the correct way. >> >> Note that some applications will check the number of cores they are running >> on and start by OpenMP (not Open MPI) as many threads as cores are found. >> Could this be the case for your application too? >> >> -- Reuti >> >> >>> Or other example: >>> My job took say 6 cpus on slave07 and 14 on slave06 but nothing was >>> running on 06 and therefore a waste of ressource on 06 and overload on >>> 07 becomes highly possible (the numbers are made up). >>> If i ran 1 Cpus in many independent jobs that would not be an issue, but >>> imagine i now request 60 cpus on slave07, that would seriously overload >>> the node in many cases. >>> >>> Or other example: >>> if i ask for say 50 CPUs, the job will start on one node, e.g, >>> slave01, but only reserving say 15 CPUs out of 64 and reserve the rest >>> on many other nodes (obviously wasting space doing nothing). >>> This has the bad consequence of allocating many more CPUs than available >>> when many jobs are running, imagine you have 10 jobs like this one... >>> some nodes will run maybe 3 even if they only have 24 CPUs... >>> >>> I hope that i have made clear what the issue is. >>> >>> I also see that the `qstat` and `qstat -f` are in disagreement. The >>> latter is correct, i checked the processes running on the nodes. >>> >>> >>> Did somebody already encounter such a problem? Does somebody have an >>> idea where to look into or what to test? >>> >>> With kind regards, ulrich >>> >>> >>> >>> <qhost.txt><qconf-sconf.txt><qconf-mp-orte.txt><qconf-all.q>_______________________________________________ >>> users mailing list >>> [email protected] >>> https://gridengine.org/mailman/listinfo/users >> > _______________________________________________ > users mailing list > [email protected] > https://gridengine.org/mailman/listinfo/users _______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
