I think the issue is that if you *dont* specifically use
pthread_attr_setstacksize the pthread library will (can?) give
each thread a stack of size equal to the stacksize rlimit.
You are correct - this is not specifically an Open MPI issue
although if it is Open MPI spawning the threads, maybe it
should be actively setting pthread_attr_setstacksize. But there
is an unfortunate tendency for users to set very large (or
unlimited) stacksize without realising these sort of repercussions.
Try catching/looking at the stacktrace of an application that is in
infinite recursion with a 1GB+ stacksize after it finally gets a
SEGV ...
Jeff Squyres wrote:
I can't think of what OMPI would be doing related to the predefined
stack size -- I am not aware of anywhere in the code where we look up
the predefine stack size and then do something with it.
That being said, I don't know what the OS and resource consumption
effects are of setting 1GB+ stack size on *any* application... Have you
tried non-MPI examples, potentially with applications as large as MPI
applications but without the complexity of MPI?
On Nov 19, 2009, at 3:13 PM, David Singleton wrote:
Depending on the setup, threads often get allocated a thread local
stack with size equal to the stacksize rlimit. Two threads maybe?
David
Terry Dontje wrote:
> A couple things to note. First Sun MPI 8.2.1 is effectively OMPI
> 1.3.4. I also reproduced the below issue using a C code so I think
this
> is a general issue with OMPI and not Fortran based.
>
> I did a pmap of a process and there were two anon spaces equal to the
> stack space set by ulimit.
>
> In one case (setting 102400) the anon spaces were next to each other
> prior to all the loadable libraries. In another case (setting 1024000)
> one anon space was locate in the same area as the first case but the
> second space was deep into some memory used by ompi.
>
> Is any of this possibly related to the predefined handles? Though I am
> not sure why it would expand based on stack size?.
>
> --td
>> Date: Thu, 19 Nov 2009 19:21:46 +0100
>> From: Paul Kapinos <kapi...@rz.rwth-aachen.de>
>> Subject: [OMPI users] exceedingly virtual memory consumption of MPI
>> environment if higher-setting "ulimit -s"
>> To: Open MPI Users <us...@open-mpi.org>
>> Message-ID: <4b058cba.3000...@rz.rwth-aachen.de>
>> Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"
>>
>> Hi volks,
>>
>> we see an exeedingly *virtual* memory consumtion through MPI processes
>> if "ulimit -s" (stack size)in profile configuration was setted higher.
>>
>> Furthermore we believe, every mpi process started, wastes about the
>> double size of `ulimit -s` value which will be set in a fresh console
>> (that is, the value is configurated in e.g. .zshenv, *not* the value
>> actually setted in the console from which the mpiexec runs).
>>
>> Sun MPI 8.2.1, an empty mpi-HelloWorld program
>> ! either if running both processes on the same host..
>>
>> .zshenv: ulimit -s 10240 --> VmPeak: 180072 kB
>> .zshenv: ulimit -s 102400 --> VmPeak: 364392 kB
>> .zshenv: ulimit -s 1024000 --> VmPeak: 2207592 kB
>> .zshenv: ulimit -s 2024000 --> VmPeak: 4207592 kB
>> .zshenv: ulimit -s 20240000 --> VmPeak: 39.7 GB!!!!
>> (see the attached files; the a.out binary is a mpi helloworld program
>> running an never ending loop).
>>
>>
>>
>> Normally, the stack size ulimit is set to some 10 MB by us, but we see
>> a lot of codes which needs *a lot* of stack space, e.g. Fortran codes,
>> OpenMP codes (and especially fortran OpenMP codes). Users tends to
>> hard-code the setting-up the higher value for stack size ulimit.
>>
>> Normally, the using of a lot of virtual memory is no problem, because
>> there is a lot of this thing :-) But... If more than one person is
>> allowed to work on a computer, you have to divide the ressources in
>> such a way that nobody can crash the box. We do not know how to limit
>> the real RAM used so we need to divide the RAM by means of setting
>> virtual memory ulimit (in our batch system e.g.. That is, for us
>> "virtual memory consumption" = "real memory consumption".
>> And real memory is not that way cheap than virtual memory.
>>
>>
>> So, why consuming the *twice* amount of stack size for each process?
>>
>> And, why consuming the virtual memory at all? We guess this virtual
>> memory is allocated for the stack (why else it will be related to the
>> stack size ulimit). But, is such allocation really needed? Is there a
>> way to avoid the vaste of virtual memory?
>>
>> best regards,
>> Paul Kapinos
>>
>>
>>
>>
>>
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
--------------------------------------------------------------------------
Dr David Singleton ANU Supercomputer Facility
HPC Systems Manager and NCI National Facility
david.single...@anu.edu.au Leonard Huxley Bldg (No. 56)
Phone: +61 2 6125 4389 Australian National University
Fax: +61 2 6125 8199 Canberra, ACT, 0200, Australia
--------------------------------------------------------------------------
