Gabriel Genellina wrote:
En Mon, 09 Nov 2009 16:05:31 -0300, Eyal Gordon <[email protected]> escribió:background:we are using python 2.4.3 on CentOS 5.3 with many threads - and our shell'sdefault stack size limit is set to 10240KB (i.e. ~10MB). we noticed that python's Threading module appears to create threads with this value as their stack size (we ran a sample program that creates 10 threads and measured its virtual memory size, then reduced the stack size limit of the shell to 5120KB - and saw that the program's virtual memory size was reduced by ~50MBs). the problem:our program uses numerous threads, and thus the virtual memory size gets to be very large. we would like to reduce the size of the stack to reduce thissize. we were looking for information about recommendation for the stack size to use, but found none.You can set the thread stack size (for threads that are going to be created, not existing threads) using threading.stack_size(SIZE) http://docs.python.org/library/threading.html#threading.stack_size
Sadly for the OP, that capability was introduced in Python 2.5. Prior to that, the only way to adjust thread stack size is via a compile time option (THREAD_STACK_SIZE). In the absence of that compile time option, the platform default is used - on pthread systems (incl Linux) refer to the manpage for pthread_attr_setstacksize() for more info, but I believe what the OP reports could reasonably be expected for his platform. The threading.stack_size() support would not be hard to backport to Python 2.4.
questions: 1. is there some rule-of-thumb for the recommended stack size for python programs of various sorts?
There is no rule of thumb because there are too many variables amongst
the supported platforms. Extensive testing would be required to
validate any selected size.
I would suggest starting with 1MB and working down. On a 32bit platform
I would suggest that 64kb might be adequate for child threads but most
likely not for the primary thread (the thread that the Python
interpreter starts in). If your app uses regexes, your stack space
requirements are likely to be larger.
FWIW the default thread stack size on Win32 is 1MB which anecdotally
seems sufficient for a wide variety of applications on that platform.
{...}
4. would the size of the stacks (which are probably not really allocated bythe linux virtual memory sub-system, unless used) have a noticeableperformance effect on a python program? same question regarding the use of alarge number of threads?I think it doesn't matter, unless you create so many threads as to exhaust the available addressing space (in 32bits, 4GB address space and 10MB per thread means 400 threads maximum).
The performance of course will degrade once the committed memory in the active working set exceeds the available real memory, as the OS will start swapping. How the OS treats unused stack space will affect that threshold. -- ------------------------------------------------------------------------- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [email protected] (pref) | Snail: PO Box 370 [email protected] (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia -- http://mail.python.org/mailman/listinfo/python-list
