Re: [OMPI devel] large virtual memory consumption on smp nodes and gridengine problems

2007-06-21 Thread Jeff Squyres
On Jun 17, 2007, at 7:32 AM, Markus Daene wrote: 3. I think it will not be possible to use the OpenFabrics kernel/user stack. The machine was installed by SUN, it seems that they did not use the one from OpenFabrics. I guess it will be a hard discussion to change this and we cannot do this b

Re: [OMPI devel] large virtual memory consumption on smp nodes and gridengine problems

2007-06-17 Thread Markus Daene
Hi Jeff, thanks for your comments. 1. I will report this to the GE mailing list. 2. We have a cluster of 18 nodes with 16 cores each (8x dual core Opteron). So we plan to run something between 1...128 processes in total, 16 per node. Of course, if the sm component allocates 512MB x 16 on on nod

Re: [OMPI devel] large virtual memory consumption on smp nodes and gridengine problems

2007-06-17 Thread Markus Daene
Hi Ralph, many thanks. This is exactly what I need. Markus Ralph Castain wrote: > Hi Markus > > There are two MCA params that can help you, I believe: > > 1. You to set the maximum size of the shared memory file with > > -mca mpool_sm_max_size xxx > > where xxx is the maximum memory file you want

Re: [OMPI devel] large virtual memory consumption on smp nodes and gridengine problems

2007-06-11 Thread Pak Lui
Markus Daene wrote: then it becomes ever worse: openmpi nicely report the (max./act.) used virtual memory to the grid engine as sum of all processes. This value is the compared with the one the user has specified with the h_vmem option, but the gridengine takes this value per process for the allo

Re: [OMPI devel] large virtual memory consumption on smp nodes and gridengine problems

2007-06-11 Thread Jeff Squyres
In addition to what Ralph said, I have the following random comments: 1. You'll have to ask on the GE mailing lists about the GE issues (2gb vs. 2000mb, etc.); I doubt we'll be of much help here on this list. 2. Do you have a very large SMP machine (i.e., 16 cores or more)? More specifical

Re: [OMPI devel] large virtual memory consumption on smp nodes and gridengine problems

2007-06-10 Thread Ralph Castain
Hi Markus There are two MCA params that can help you, I believe: 1. You to set the maximum size of the shared memory file with -mca mpool_sm_max_size xxx where xxx is the maximum memory file you want, expressed in bytes. The default value I see is 512MBytes. 2. You can set the size/peer of the