Re: [mp2] Memory Leak

[Stas Bekman]

Pringle, Chris (HP-PSG) wrote:

You will find a few comments regarding your code below:

sub handler 
{
	# Get the filter object
	my($f,$bb) = @_;

	my $c = $f->c;
You probably don't need $c, you seem to be in the request filter...

        # Declare a buffer
        my($buffer) = "";
        my($scratch) = "";
        # Only done on the FIRST pass of the filter
        unless($f->ctx)
        {
                $f->r->headers_out->unset('Content-Length');
                $f->ctx(1);
                # Ensure there is a buffer variable in the filter
context
                $f->ctx({buffer => ''});
        }
BTW, you don't need $f->ctx(1); if you then immediately set it to something 
else. Also you don't need that extra redirection, it'll be a bit faster to 
just store the buffer as is:
  $f->ctx(''}

and change the condition to:

  unless (defined $f->ctx)

and later on:

$f->ctx($buffer} if defined $buffer;

        # Read as much data as there is available and put it into
$buffer
        while($f->read($scratch, BUFF_LEN))
        {
                # If buffer > 500K print buffer and stop
                if(length($buffer) > 500000)
                {
                        $f->print($buffer);
                        return Apache::DECLINED;
                }
                else
                {
                        $buffer = $buffer . $scratch;
                }
this will eat a lot of memory (though it'll be reused), better use .=.

% perl-5.8.3-ithread ~/bin/size_it.pl '$chunk = "x" x 4096; my $buffer = ""; 
$buffer = $buffer . $chunk for 1..10'
resident   :  124k
rss        :  128k
share      :    0k
size       :  124k
vsize      :  132k

% perl-5.8.3-ithread ~/bin/size_it.pl '$chunk = "x" x 4096; my $buffer = ""; 
$buffer .= $chunk for 1..10'
resident   :   44k
rss        :   48k
share      :    0k
size       :   44k
vsize      :  132k

Also it's much faster on the latest perls:

% perl-5.8.3 /tmp/append
          Rate concat append
concat  5244/s     --   -86%
append 38314/s   631%     --
there was not much difference in earlier perls (only about 7-105). I think 
there is some bug in 5.8.3.

Here is the benchmark I've used:

use Benchmark qw(:all) ;

my $chunk = "x" x 4096;

cmpthese(100000, {
    append  => \&append,
    concat  => \&concat,
});
sub append {
    my $buffer = '';
    for (1..10) {
        $buffer .= $chunk;
    }
}
sub concat {
    my $buffer = '';
    for (1..10) {
        $buffer = $buffer . $chunk;
    }
}


Remember that f->ctx doesn't copy anything, so if your code is efficient, the 
buffering up data won't an additional cost memory and time.



        # If this is the last bucket
        if($f->seen_eos)
        {
                #
                # Do some filtering here ....
                #
                #
                $f->print($buffer);
        }
        $f->ctx({ buffer => $buffer});
also you don't want to store anything on the last invocation (as it's of no 
use), so put it into the else branch:

if ($f->seen_eos) {
 ...
}
else {
  $f->ctx({ buffer => $buffer});
}
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html