To tell you all what noone wanted to tell me, yes, it does seem to be the memory manager. Compiling everything with --with-memory-manager=none returns the vmem use to the more reasonable ~100MB per process (down from >8GB).
I take it this may affect my peak bandwidth over infiniband. What's the general feeling about how bad this is? On Tue, 2008-05-13 at 13:12 +1000, Terry Frankcombe wrote: > Hi folks > > I'm trying to run an MPI app on an infiniband cluster with OpenMPI > 1.2.6. > > When run on a single node, this app is grabbing large chunks of memory > (total per process ~8.5GB, including strace showing a single 4GB grab) > but not using it. The resident memory use is ~40MB per process. When > this app is compiled in serial mode (with conditionals to remove the MPI > calls) the memory use is more like what you'd expect, 40MB res and > ~100MB vmem. > > Now I didn't write it so I'm not sure what extra stuff the MPI version > does, and we haven't tracked down the large memory grabs. > > Could it be that this vmem is being grabbed by the OpenMPI memory > manager rather than directly by the app? > > Ciao > Terry > >
