I ran my baseline test with 500k requests, and got the following:
(Note that all the figures will have an error of +/- 0.1)
baseline 500k requests 1.7%
So it would seem that there is not a specific problem in readline, or my
test case is messed up. FYI here are my 2 handlers:
def baseline_handler(req):
req.content_type = 'text/plain'
req.write('ok baseline:')
return apache.OK
def readline_handler(req):
# the body of the request consists of
# '\n'.join([ 'a'*10 for i in xrange(0,10) ])
req.content_type = 'text/plain'
count = 0
while(1):
line = req.readline()
if not line:
break
count += 1
req.write('ok readline: %d lines read' % count)
return apache.OK
Jim
Jim Gallacher wrote:
> I'll have some time to investigate this over the next couple of days. I
> ran my leaktest script for FieldStorage and readline, and FieldStorage
> certainly still leaks, but I'm not so sure about readline itself.
>
> baseline 1k requests 1.2%
> readline 500k requests 1.6%
> fieldstorage 498k requests 10.1%
>
> The memory consumption figures are for a machine with 512MB ram.
>
> I'm running my baseline test with 500k requests right now to see if the
> 1.6% figure for readline represents a real leak in that function, or if
> it is just mod_python itself.
>
> My memory leak test suite is probably at the point that other people
> will find it useful. Once I've written a README explaining its use I'll
> commit it to the repository so everybody to play. If you anyone wants to
> give it a shot in the interim I can email it to you. Give me shout
> offlist.
>
> I haven't had a chance to look at the code you highlight below, or at
> least not closely. The whole req_readline function looks like it will
> require a good strong cup of coffee to fully comprehend. ;)
>
> Jim
>
> Alexis Marrero wrote:
>> Experimenting on this issue, I noticed that neither of the following set
>> of "ifs" are ever met:
>>
>>
>> 786 /* Free old rbuff as the old contents have been copied over and
>> 787 we are about to allocate a new rbuff. Perhaps this could
>> be reused
>> 788 somehow? */
>> 789 if (self->rbuff_pos >= self->rbuff_len && self->rbuff != NULL)
>> 790 {
>> 791 free(self->rbuff);
>> 792 self->rbuff = NULL;
>> 793 }
>>
>>
>> --------
>>
>> 846 /* Free rbuff if we're done with it */
>> 847 if (self->rbuff_pos >= self->rbuff_len && self->rbuff != NULL)
>> 848 {
>> 849 free(self->rbuff);
>> 850 self->rbuff = NULL;
>> 851 }
>>
>> I noticed that by putting some statements to write to the output
>> stream. They never execute.
>>
>> /amn
>>
>> On Aug 10, 2006, at 1:43 PM, Alexis Marrero wrote:
>>
>>> All,
>>>
>>> We are trying to nail down a memory leak that happens only when
>>> documents are POSTed to the server.
>>>
>>> For testing we have a short script that does:
>>>
>>> while True:
>>> dictionary_of_parameters = {'field1': 'a'*100000}
>>> post('url...', dictionary_of_parameters)
>>>
>>> Then we run "top" on the server and watch the server memory grow
>>> without bound. Why do we know that the problem is in
>>> request.readline()? If I go to
>>> mod_python.util.FieldStorage.read_to_boundary() and add the following
>>> statement:
>>>
>>> def read_to_boundary(...):
>>> return True
>>> ...
>>>
>>> as the first executable line in the function the memory does not grow.
>>>
>>> I have read the req_readline a 1000 time and I can't figure out where
>>> the problem is.
>>>
>>>
>>> My config:
>>> Python 2.4.1
>>> mod_python 3.2.10
>>>
>>> Our request handler does nothing other than using
>>> util.FieldStorage(req) and req.write('hello').
>>>
>>> I have some suspicion that it has to do with:
>>> ....
>>> 19 * requestobject.c
>>> 20 *
>>> 21 * $Id: requestobject.c 420297 2006-07-09 13:53:06Z nlehuen $
>>> 22 *
>>> 23 */
>>> ....
>>> 846 /* Free rbuff if we're done with it */
>>> 847 if (self->rbuff_pos >= self->rbuff_len && self->rbuff !=
>>> NULL)
>>> 848 {
>>> 849 free(self->rbuff);
>>> 850 self->rbuff = NULL;
>>> 851 }
>>> 852
>>>
>>> Though, I can't confirm.
>>>
>>>
>>> /amn
>>>
>>
>
>
