Gabriel Genellina wrote:
En Mon, 09 Nov 2009 16:05:31 -0300, Eyal Gordon <eyal.gor...@gmail.com>
escribió:
background:
we are using python 2.4.3 on CentOS 5.3 with many threads - and our
shell's
default stack size limit is set to 10240KB (i.e. ~10MB).
we noticed that python's Threading module appears to create threads with
this value as their stack size (we ran a sample program that creates 10
threads and measured its virtual memory size, then reduced the stack size
limit of the shell to 5120KB - and saw that the program's virtual memory
size was reduced by ~50MBs).
the problem:
our program uses numerous threads, and thus the virtual memory size
gets to
be very large. we would like to reduce the size of the stack to reduce
this
size. we were looking for information about recommendation for the stack
size to use, but found none.
You can set the thread stack size (for threads that are going to be
created, not existing threads) using threading.stack_size(SIZE)
http://docs.python.org/library/threading.html#threading.stack_size
Sadly for the OP, that capability was introduced in Python 2.5. Prior
to that, the only way to adjust thread stack size is via a compile time
option (THREAD_STACK_SIZE). In the absence of that compile time option,
the platform default is used - on pthread systems (incl Linux) refer to
the manpage for pthread_attr_setstacksize() for more info, but I believe
what the OP reports could reasonably be expected for his platform.
The threading.stack_size() support would not be hard to backport to
Python 2.4.
questions:
1. is there some rule-of-thumb for the recommended stack size for python
programs of various sorts?
There is no rule of thumb because there are too many variables amongst
the supported platforms. Extensive testing would be required to
validate any selected size.
I would suggest starting with 1MB and working down. On a 32bit platform
I would suggest that 64kb might be adequate for child threads but most
likely not for the primary thread (the thread that the Python
interpreter starts in). If your app uses regexes, your stack space
requirements are likely to be larger.
FWIW the default thread stack size on Win32 is 1MB which anecdotally
seems sufficient for a wide variety of applications on that platform.
{...}
4. would the size of the stacks (which are probably not really
allocated by
the linux virtual memory sub-system, unless used) have a noticeable
performance effect on a python program? same question regarding the
use of a
large number of threads?
I think it doesn't matter, unless you create so many threads as to exhaust
the available addressing space (in 32bits, 4GB address space and 10MB per
thread means 400 threads maximum).
The performance of course will degrade once the committed memory in the
active working set exceeds the available real memory, as the OS will
start swapping. How the OS treats unused stack space will affect that
threshold.
--
-------------------------------------------------------------------------
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: andy...@bullseye.apana.org.au (pref) | Snail: PO Box 370
andy...@pcug.org.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia
--
http://mail.python.org/mailman/listinfo/python-list
