I'll have some time to investigate this over the next couple of days. I
ran my leaktest script for FieldStorage and readline, and FieldStorage
certainly still leaks, but I'm not so sure about readline itself.
baseline 1k requests 1.2%
readline 500k requests 1.6%
fieldstorage 498k requests 10.1%
The memory consumption figures are for a machine with 512MB ram.
I'm running my baseline test with 500k requests right now to see if the
1.6% figure for readline represents a real leak in that function, or if
it is just mod_python itself.
My memory leak test suite is probably at the point that other people
will find it useful. Once I've written a README explaining its use I'll
commit it to the repository so everybody to play. If you anyone wants to
give it a shot in the interim I can email it to you. Give me shout
offlist.
I haven't had a chance to look at the code you highlight below, or at
least not closely. The whole req_readline function looks like it will
require a good strong cup of coffee to fully comprehend. ;)
Jim
Alexis Marrero wrote:
> Experimenting on this issue, I noticed that neither of the following set
> of "ifs" are ever met:
>
>
> 786 /* Free old rbuff as the old contents have been copied over and
> 787 we are about to allocate a new rbuff. Perhaps this could
> be reused
> 788 somehow? */
> 789 if (self->rbuff_pos >= self->rbuff_len && self->rbuff != NULL)
> 790 {
> 791 free(self->rbuff);
> 792 self->rbuff = NULL;
> 793 }
>
>
> --------
>
> 846 /* Free rbuff if we're done with it */
> 847 if (self->rbuff_pos >= self->rbuff_len && self->rbuff != NULL)
> 848 {
> 849 free(self->rbuff);
> 850 self->rbuff = NULL;
> 851 }
>
> I noticed that by putting some statements to write to the output
> stream. They never execute.
>
> /amn
>
> On Aug 10, 2006, at 1:43 PM, Alexis Marrero wrote:
>
>> All,
>>
>> We are trying to nail down a memory leak that happens only when
>> documents are POSTed to the server.
>>
>> For testing we have a short script that does:
>>
>> while True:
>> dictionary_of_parameters = {'field1': 'a'*100000}
>> post('url...', dictionary_of_parameters)
>>
>> Then we run "top" on the server and watch the server memory grow
>> without bound. Why do we know that the problem is in
>> request.readline()? If I go to
>> mod_python.util.FieldStorage.read_to_boundary() and add the following
>> statement:
>>
>> def read_to_boundary(...):
>> return True
>> ...
>>
>> as the first executable line in the function the memory does not grow.
>>
>> I have read the req_readline a 1000 time and I can't figure out where
>> the problem is.
>>
>>
>> My config:
>> Python 2.4.1
>> mod_python 3.2.10
>>
>> Our request handler does nothing other than using
>> util.FieldStorage(req) and req.write('hello').
>>
>> I have some suspicion that it has to do with:
>> ....
>> 19 * requestobject.c
>> 20 *
>> 21 * $Id: requestobject.c 420297 2006-07-09 13:53:06Z nlehuen $
>> 22 *
>> 23 */
>> ....
>> 846 /* Free rbuff if we're done with it */
>> 847 if (self->rbuff_pos >= self->rbuff_len && self->rbuff !=
>> NULL)
>> 848 {
>> 849 free(self->rbuff);
>> 850 self->rbuff = NULL;
>> 851 }
>> 852
>>
>> Though, I can't confirm.
>>
>>
>> /amn
>>
>
>
