How about using the exclusive complex scheduling feature? http://docs.oracle.com/cd/E24901_01/doc.62/e21978/management.htm#sthref431
And Reuti's reply to a similar problem: http://gridengine.org/pipermail/users/2012-May/003535.html -Ron ----- Original Message ----- From: Joseph Farran <jfar...@uci.edu> To: "users@gridengine.org" <users@gridengine.org> Cc: Sent: Friday, June 8, 2012 9:15 PM Subject: [gridengine users] Understanding Parallel Enviroment ( whole nodes ) Greetings. I am try to setup my MPI Parallel Environment so that whole nodes are used before going to the next node when looking for cores. Our nodes have 64 cores. What I like is that if I ask for 128 cores (slots), one compute node is selected with 64 cores, and then the next one with 64 cores. At the suggestion of Prakashan I setup all of my 64 cores nodes with: qconf -rattr exechost complex_values "slots=64" node To hopefully tell OGE to use 64-cores per nodes and no more. Using a simple Parallel Environment called "mpi" with "$fill_up" allocation rule, I am getting weird results. When I ask for 128 cores with: #$ -pe mpi 128 Some times I get two nodes at 64-core each which is correct: PE_HOSTFILE file: (/var/spool/oge/compute-2-5/active_jobs/83.1/pe_hostfile) compute-2-5.local 64 all.q@compute-2-5.local UNDEFINED compute-2-7.local 64 all.q@compute-2-7.local UNDEFINED Other times, I get sporadic mixture that I can't make sense of, like this one with 3 nodes at 54, 64 and 10 cores: PE_HOSTFILE file: (/var/spool/oge/compute-1-2/active_jobs/84.1/pe_hostfile) compute-1-2.local 54 all.q@compute-1-2.local UNDEFINED compute-2-4.local 64 all.q@compute-2-4.local UNDEFINED compute-2-6.local 10 all.q@compute-2-6.local UNDEFINED What setting is causing this and/or load sensor? What I am looking for is to fill up one 64-core node before it goes on to the next one. So that if I ask for 128 cores, I will always get 2 whole nodes. _______________________________________________ users mailing list users@gridengine.org https://gridengine.org/mailman/listinfo/users _______________________________________________ users mailing list users@gridengine.org https://gridengine.org/mailman/listinfo/users
