[OMPI users] over-subscription of cores

2011-12-23 Thread Santosh Ansumali
Dear All, We are running a PDE solver which is memory bound. Due to cache related issue, smaller number of grid point per core leads to better performance for this code. Thus, though available memory per core is more than 2 GB, we are able to good performance by using less than 1 GB

Re: [OMPI users] over-subscription of cores

2011-12-23 Thread Gustavo Correa
I don't know about the grid engine/ SGE. However, in Torque, the batch/resource manager I use, to allow oversubscription, you need to modify the batch server nodes file and pretend the nodes have more cores than the physical ones. [Something like 'node01 np=8' would change to 'node01 np=16' for in

Re: [OMPI users] over-subscription of cores

2011-12-26 Thread Reuti
Am 23.12.2011 um 21:16 schrieb Gustavo Correa: > I don't know about the grid engine/ SGE. > However, in Torque, the batch/resource manager I use, > to allow oversubscription, you need to modify the batch server nodes file > and pretend the nodes have more cores than the physical ones. > [Somethin

Re: [OMPI users] over-subscription of cores

2011-12-26 Thread Santosh Ansumali
Dear Dr. Correa, Sorry for my ignorance on cluster maintenance. So far our cluster is just set-up by a vendor and we do not know more details. So far I am understanding the concept but we are not able to follow what precisely we need to try out for allowing oversubscription. In this submissio

Re: [OMPI users] over-subscription of cores

2011-12-26 Thread Matthieu Brucher
Hi, If your problem is memory bound and if you don't use the whole memory capacity of one node, it means that you are limited by your memory bandwidth. In this case oversubscribing the number of processes will lead to worse behavior, as all processes will fight for the same memory bandwidth. Just

Re: [OMPI users] over-subscription of cores

2011-12-26 Thread Santosh Ansumali
Thanks for the response. May be I am wrong. However my argument is as follows: our test shows that a 100^3 grid per core performs 10 times faster (normalised in proper unit) than 200^3. Both of these sizes are not fitting in cache. 100^3 run is benefiting due to smaller size where compiler is gu

Re: [OMPI users] over-subscription of cores

2011-12-26 Thread Ralph Castain
I confess I'm confused. OMPI allows you to "oversubscribe" a node without any modification of job allocations. Just ask it to launch however many processes you want - it will ignore the allocated number of slots and do it. It will set the sched_yield appropriately to deal with the oversubscripti

Re: [OMPI users] over-subscription of cores

2011-12-26 Thread Reuti
Hi, Am 26.12.2011 um 17:55 schrieb Santosh Ansumali: > Dear Dr. Correa, >Sorry for my ignorance on cluster maintenance. So far our > cluster is just set-up by a vendor and we do not know more details. > So far I am understanding the concept but we are not able to follow > what precisely we