A couple things to note. First Sun MPI 8.2.1 is effectively OMPI
1.3.4. I also reproduced the below issue using a C code so I think this
is a general issue with OMPI and not Fortran based.
I did a pmap of a process and there were two anon spaces equal to the
stack space set by ulimit.
In one case (setting 102400) the anon spaces were next to each other
prior to all the loadable libraries. In another case (setting 1024000)
one anon space was locate in the same area as the first case but the
second space was deep into some memory used by ompi.
Is any of this possibly related to the predefined handles? Though I am
not sure why it would expand based on stack size?.
--td
Date: Thu, 19 Nov 2009 19:21:46 +0100
From: Paul Kapinos <kapi...@rz.rwth-aachen.de>
Subject: [OMPI users] exceedingly virtual memory consumption of MPI
environment if higher-setting "ulimit -s"
To: Open MPI Users <us...@open-mpi.org>
Message-ID: <4b058cba.3000...@rz.rwth-aachen.de>
Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"
Hi volks,
we see an exeedingly *virtual* memory consumtion through MPI processes
if "ulimit -s" (stack size)in profile configuration was setted higher.
Furthermore we believe, every mpi process started, wastes about the
double size of `ulimit -s` value which will be set in a fresh console
(that is, the value is configurated in e.g. .zshenv, *not* the value
actually setted in the console from which the mpiexec runs).
Sun MPI 8.2.1, an empty mpi-HelloWorld program
! either if running both processes on the same host..
.zshenv: ulimit -s 10240 --> VmPeak: 180072 kB
.zshenv: ulimit -s 102400 --> VmPeak: 364392 kB
.zshenv: ulimit -s 1024000 --> VmPeak: 2207592 kB
.zshenv: ulimit -s 2024000 --> VmPeak: 4207592 kB
.zshenv: ulimit -s 20240000 --> VmPeak: 39.7 GB!!!!
(see the attached files; the a.out binary is a mpi helloworld program
running an never ending loop).
Normally, the stack size ulimit is set to some 10 MB by us, but we see a
lot of codes which needs *a lot* of stack space, e.g. Fortran codes,
OpenMP codes (and especially fortran OpenMP codes). Users tends to
hard-code the setting-up the higher value for stack size ulimit.
Normally, the using of a lot of virtual memory is no problem, because
there is a lot of this thing :-) But... If more than one person is
allowed to work on a computer, you have to divide the ressources in such
a way that nobody can crash the box. We do not know how to limit the
real RAM used so we need to divide the RAM by means of setting virtual
memory ulimit (in our batch system e.g.. That is, for us
"virtual memory consumption" = "real memory consumption".
And real memory is not that way cheap than virtual memory.
So, why consuming the *twice* amount of stack size for each process?
And, why consuming the virtual memory at all? We guess this virtual
memory is allocated for the stack (why else it will be related to the
stack size ulimit). But, is such allocation really needed? Is there a
way to avoid the vaste of virtual memory?
best regards,
Paul Kapinos
