Jim,
I found the culprit!!!
There are two unrelated memory leaks.
The first one is in req_readline().
This code:
/* is there anything left in the rbuff from previous reads? */
if (self->rbuff_pos < self->rbuff_len) {
/* if yes, process that first */
while (self->rbuff_pos < self->rbuff_len) {
buffer[copied++] = self->rbuff[self->rbuff_pos];
if ((self->rbuff[self->rbuff_pos++] == '\n') ||
(copied == len)) {
/* our work is done */
/* resize if necessary */
if (copied < len)
if(_PyString_Resize(&result, copied))
return NULL;
return result;
}
}
}
Should look like this:
/* is there anything left in the rbuff from previous reads? */
if (self->rbuff_pos < self->rbuff_len) {
/* if yes, process that first */
while (self->rbuff_pos < self->rbuff_len) {
buffer[copied++] = self->rbuff[self->rbuff_pos];
if ((self->rbuff[self->rbuff_pos++] == '\n') ||
(copied == len)) {
/* our work is done */
/* resize if necessary */
if (copied < len)
if(_PyString_Resize(&result, copied))
return NULL;
if (self->rbuff_pos >= self->rbuff_len && self->rbuff !=
NULL)
{
free(self->rbuff);
self->rbuff = NULL;
}
return result;
}
}
}
That solves one. Like I mentioned in one of the emails to the mailing
list, the buffer was not been freed in the last readline().
The second one, for which I don't have a fix yet is apache.make_table()
in mod_python/util.py line 152. If I comment lines 152, 225, 227 you
will see that memory doesn't grow. I will keep investigating...
Until the next email.
/amn
Jim Gallacher wrote:
I ran my baseline test with 500k requests, and got the following:
(Note that all the figures will have an error of +/- 0.1)
baseline 500k requests 1.7%
So it would seem that there is not a specific problem in readline, or my
test case is messed up. FYI here are my 2 handlers:
def baseline_handler(req):
req.content_type = 'text/plain'
req.write('ok baseline:')
return apache.OK
def readline_handler(req):
# the body of the request consists of
# '\n'.join([ 'a'*10 for i in xrange(0,10) ])
req.content_type = 'text/plain'
count = 0
while(1):
line = req.readline()
if not line:
break
count += 1
req.write('ok readline: %d lines read' % count)
return apache.OK
Jim
Jim Gallacher wrote:
I'll have some time to investigate this over the next couple of days. I
ran my leaktest script for FieldStorage and readline, and FieldStorage
certainly still leaks, but I'm not so sure about readline itself.
baseline 1k requests 1.2%
readline 500k requests 1.6%
fieldstorage 498k requests 10.1%
The memory consumption figures are for a machine with 512MB ram.
I'm running my baseline test with 500k requests right now to see if the
1.6% figure for readline represents a real leak in that function, or if
it is just mod_python itself.
My memory leak test suite is probably at the point that other people
will find it useful. Once I've written a README explaining its use I'll
commit it to the repository so everybody to play. If you anyone wants to
give it a shot in the interim I can email it to you. Give me shout
offlist.
I haven't had a chance to look at the code you highlight below, or at
least not closely. The whole req_readline function looks like it will
require a good strong cup of coffee to fully comprehend. ;)
Jim
Alexis Marrero wrote:
Experimenting on this issue, I noticed that neither of the following set
of "ifs" are ever met:
786 /* Free old rbuff as the old contents have been copied over and
787 we are about to allocate a new rbuff. Perhaps this could
be reused
788 somehow? */
789 if (self->rbuff_pos >= self->rbuff_len && self->rbuff != NULL)
790 {
791 free(self->rbuff);
792 self->rbuff = NULL;
793 }
--------
846 /* Free rbuff if we're done with it */
847 if (self->rbuff_pos >= self->rbuff_len && self->rbuff != NULL)
848 {
849 free(self->rbuff);
850 self->rbuff = NULL;
851 }
I noticed that by putting some statements to write to the output
stream. They never execute.
/amn
On Aug 10, 2006, at 1:43 PM, Alexis Marrero wrote:
All,
We are trying to nail down a memory leak that happens only when
documents are POSTed to the server.
For testing we have a short script that does:
while True:
dictionary_of_parameters = {'field1': 'a'*100000}
post('url...', dictionary_of_parameters)
Then we run "top" on the server and watch the server memory grow
without bound. Why do we know that the problem is in
request.readline()? If I go to
mod_python.util.FieldStorage.read_to_boundary() and add the following
statement:
def read_to_boundary(...):
return True
...
as the first executable line in the function the memory does not grow.
I have read the req_readline a 1000 time and I can't figure out where
the problem is.
My config:
Python 2.4.1
mod_python 3.2.10
Our request handler does nothing other than using
util.FieldStorage(req) and req.write('hello').
I have some suspicion that it has to do with:
....
19 * requestobject.c
20 *
21 * $Id: requestobject.c 420297 2006-07-09 13:53:06Z nlehuen $
22 *
23 */
....
846 /* Free rbuff if we're done with it */
847 if (self->rbuff_pos >= self->rbuff_len && self->rbuff !=
NULL)
848 {
849 free(self->rbuff);
850 self->rbuff = NULL;
851 }
852
Though, I can't confirm.
/amn
