Re: multythreading app memory consumption
Bryan Olson wrote: In Python 2.5, each thread will be allocated thread.stack_size() bytes of stack address space. Note that address space is not physical memory, nor even virtual memory. On modern operating systems, the memory gets allocated as needed, and 150 threads is not be a problem. Just a note that [thread|threading].stack_size() returns 0 to indicate the platform default, and that value will always be returned unless an explicit value has previously been set. The Posix thread platforms (those that support programmatic setting of this parameter) have the best support for sanity checking the requested size - the value gets checked when actually set, rather than when the thread creation is attempted. The platform default thread stack sizes I can recall are: Windows: 1MB (though this may be affected by linker options) Linux:1MB or 8MB depending on threading library and/or distro FreeBSD: 64kB -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: multythreading app memory consumption
Thank you guys for your replies. I've just realized that there was no memory leak and it was just my mistake to think so. I've almost disappointed with my favorite programming language before addressing the problem. Actually the app consume as much memory as it should and I've just miscalculated. Regards -- http://mail.python.org/mailman/listinfo/python-list
Re: multythreading app memory consumption
Dennis Lee Bieber wrote: On Mon, 23 Oct 2006 03:31:28 +0400, Roman Petrichev [EMAIL PROTECTED] declaimed the following in comp.lang.python: Hi folks. I've just faced with very nasty memory consumption problem. I have a multythreaded app with 150 threads which use the only and the same function - through urllib2 it just gets the web page's html code and assigns it to local variable. On the next turn the variable is overritten with another page's code. At every moment the summary of values of the variables containig code is not more than 15Mb (I've just invented a tricky way to measure this). But during the first 30 minutes all the system memory (512Mb) is consumed and 'MemoryError's is arising. Why is it so and how can I limit the memory consumption in borders, say, 400Mb? Maybe there is a memory leak there? Thnx How much stack space gets allocated for 150 threads? Actually I don't know. How can I get to know this? Q = Queue.Queue() for i in rez: #rez length - 5000 Can't be the test code as you don't show the imports or where rez is defined. Isn't it clear that rez is just a list of 5000 urls? I cannot place it here, but believe me all of them are not big - At every moment the summary of values of the variables containig code is not more than 15Mb Regards -- http://mail.python.org/mailman/listinfo/python-list
Re: multythreading app memory consumption
Roman Petrichev wrote: try: url = Q.get() except Queue.Empty: break This code will never raise the Queue.Empty exception. Only a non-blocking get does: url = Q.get(block=False) As mentioned before you should post working code if you expect people to help. i. -- http://mail.python.org/mailman/listinfo/python-list
Re: multythreading app memory consumption
At Sunday 22/10/2006 20:31, Roman Petrichev wrote: I've just faced with very nasty memory consumption problem. I have a multythreaded app with 150 threads which use the only and the same function - through urllib2 it just gets the web page's html code and assigns it to local variable. On the next turn the variable is Consider using asyncore or the Twisted framework instead of threads. -- Gabriel Genellina Softlab SRL __ Correo Yahoo! Espacio para todos tus mensajes, antivirus y antispam ¡gratis! ¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar -- http://mail.python.org/mailman/listinfo/python-list
Re: multythreading app memory consumption
Roman Petrichev: Dennis Lee Bieber wrote: How much stack space gets allocated for 150 threads? Actually I don't know. How can I get to know this? On Linux, each thread will often be allocated 10 megabytes of stack. This can be viewed and altered with the ulimit command. Neil -- http://mail.python.org/mailman/listinfo/python-list
Re: multythreading app memory consumption
Roman Petrichev wrote:
Hi folks.
I've just faced with very nasty memory consumption problem.
I have a multythreaded app with 150 threads
[...]
The test app code:
Q = Queue.Queue()
for i in rez: #rez length - 5000
Q.put(i)
def checker():
while True:
try:
url = Q.get()
except Queue.Empty:
break
try:
opener = urllib2.urlopen(url)
data = opener.read()
opener.close()
except:
sys.stderr.write('ERROR: %s\n' % traceback.format_exc())
try:
opener.close()
except:
pass
continue
print len(data)
for i in xrange(150):
new_thread = threading.Thread(target=checker)
new_thread.start()
Don't know if this is the heart of your problem, but there's no
limit to how big data could be, after
data = opener.read()
Furthermore, you keep it until data gets over-written the next
time through the loop. You might try restructuring checker() to
make data local to one iteration, as in:
def checker():
while True:
onecheck()
def onecheck():
try:
url = Q.get()
except Queue.Empty:
break
try:
opener = urllib2.urlopen(url)
data = opener.read()
opener.close()
print len(data)
except:
sys.stderr.write('ERROR: %s\n' % traceback.format_exc())
try:
opener.close()
except:
pass
--
--Bryan
--
http://mail.python.org/mailman/listinfo/python-list
Re: multythreading app memory consumption
Dennis Lee Bieber wrote: How much stack space gets allocated for 150 threads? In Python 2.5, each thread will be allocated thread.stack_size() bytes of stack address space. Note that address space is not physical memory, nor even virtual memory. On modern operating systems, the memory gets allocated as needed, and 150 threads is not be a problem. -- --Bryan -- http://mail.python.org/mailman/listinfo/python-list
Re: multythreading app memory consumption
Gabriel Genellina wrote: At Sunday 22/10/2006 20:31, Roman Petrichev wrote: I've just faced with very nasty memory consumption problem. I have a multythreaded app with 150 threads which use the only and the same function - through urllib2 it just gets the web page's html code and assigns it to local variable. On the next turn the variable is Consider using asyncore or the Twisted framework instead of threads. But those won't play nice with urllib2. -- --Bryan -- http://mail.python.org/mailman/listinfo/python-list
multythreading app memory consumption
Hi folks.
I've just faced with very nasty memory consumption problem.
I have a multythreaded app with 150 threads which use the only and the
same function - through urllib2 it just gets the web page's html code
and assigns it to local variable. On the next turn the variable is
overritten with another page's code. At every moment the summary of
values of the variables containig code is not more than 15Mb (I've just
invented a tricky way to measure this). But during the first 30 minutes
all the system memory (512Mb) is consumed and 'MemoryError's is arising.
Why is it so and how can I limit the memory consumption in borders, say,
400Mb? Maybe there is a memory leak there?
Thnx
The test app code:
Q = Queue.Queue()
for i in rez: #rez length - 5000
Q.put(i)
def checker():
while True:
try:
url = Q.get()
except Queue.Empty:
break
try:
opener = urllib2.urlopen(url)
data = opener.read()
opener.close()
except:
sys.stderr.write('ERROR: %s\n' % traceback.format_exc())
try:
opener.close()
except:
pass
continue
print len(data)
for i in xrange(150):
new_thread = threading.Thread(target=checker)
new_thread.start()
--
http://mail.python.org/mailman/listinfo/python-list
