Re: How to debug bizarre memory corruption in mod_perl
"Stephen Clouse" <[EMAIL PROTECTED]> wrote:
> Under [mod_perl 2 and perl 5.10 on Fedora 9] I am getting some of the
> most bizarre and insidious perl core errors I've ever seen in my 15
> years of using perl.
>
> Attempt to free unreferenced scalar: SV 0xbd266be4, Perl interpreter:
> 0xba01c410 at /usr/lib/perl5/vendor_perl/5.10.0/HTML/Mason/Component.pm
> line
> 147.
> panic: attempt to copy value 20 to a freed scalar bd49dffc
> Use of freed value in iteration
> panic: free from wrong pool
>
> Not to mention countless application errors involving dereferencing.
Dereferencing demons, eh? Your error rang a bell with me. But it wasn't
until I saw List/Util/Util.so in your follow-up post that I remembered,
List::Util did that same scary crashing thing to me a while back while I
was debugging krang (which is mod_perl 1.3x-based).
It boiled down to the (IMO) senseless and unnecessary use of weak
referrences somewhere deep in the dark bowels of the Scalar-List-Utils
package. I'd read somewhere that weak references were not all that stable
in certain recent versions of perl, so when I saw all the weakening being
done in there (and all the XS bugs the module has had to fix, in its change
log:
http://search.cpan.org/src/GBARR/Scalar-List-Utils-1.19/Changes
At the time, upgrading the module to the latest release didn't stop my big
scary memory reference errors. So I read through the code that used the
module and was astonished at why the module was being used: the *one* line
of code in the entire module that I was troubleshooting that *depended* on
List::Util was a single, simple call to its exported method: first()
What does first() do? It invokes kilobytes of fast, compiled XS code to
...huh? Return to me the first element in a list I pass to it that returns
true for a code block I pass to it? Yup. Like this:
# from the POD doco at
#
http://search.cpan.org/~gbarr/Scalar-List-Utils-1.19/lib/List/Util.pm#DESCRIPTION
$foo = first { defined($_) } @list; # first defined value in @list
Who needs to install a CPAN module to do that? I personally would have
written it as:
$foo = (grep defined, @list)[0]; # first defined value in @list
But thats kind of non-obvious to the programmer who finds it later (if they
dont read the comment, that is). You could also simply replace "first"
with "grep" (and a pair of parens) and get the same effect:
($foo) = grep { defined($_) } @list; # first defined value in @list
(at the cost of one extra byte and some wasted cpu cycles if the compiler
doesn't notice that the lvalue is a list of length one, and lets grep
process the whole list anyway, even after it finds and assigns a defined
element in there).
IMO, even this would be better than loading List::Utils and calling
first():
my $foo;
for (@list) {
next unless defined($_);
$foo = $_;
last;
}
So I fixed my bug by writing a line or two of perl to in order to remove
one call to the List::Util module's first() function.
> My problem is that I don't even know where or how to start tracing or
> debugging such an issue to determine precisely what is scribbling on
> perl's
> memory.
Starting from your error message (line 147 of HTML::Mason::Component) it
appears HTML::Mason is calling a "dynamic method" method (one named in a
variable). No help there. Okay, what else is HTML::Mason::Component
using? What libraries has it loaded? Is List::Utils being used?
This looked suspicious to me:
use HTML::Mason::Tools qw(absolute_comp_path can_weaken);
Just because you "can_weaken" a reference, does that mean you should? Sure
enough HTML::Mason::Tools loads Scalar::Util apparently to do some
reference weakening.
I'd check my Mason component for use of the functions that the
Scalar-List-Utils modules provide:
List::Util
first max maxstr min minstr reduce shuffle sum
Scalar::Util
blessed dualvar isweak readonly refaddr reftype
tainted weaken isvstring looks_like_number set_prototype
And see if rewriting these in pure perl helps. Also, if it does, but would
require changing code that you'd rather not monkey with, it looks like
there is a pure-perl version of the module that can be installed by passing
a -pm option to Makefile.PL:
perl Makefile.PL -pm
make
make test
make install
Anyway, I hope somehow, some of this helps!
-dave
Re: How to debug bizarre memory corruption in mod_perl
On 9 Jul 2008, at 09:09, David Kaufman wrote:
# from the POD doco at
#
http://search.cpan.org/~gbarr/Scalar-List-Utils-1.19/lib/List/Util.pm#DESCRIPTION
$foo = first { defined($_) } @list; # first defined value in @list
Who needs to install a CPAN module to do that? I personally would
have
written it as:
$foo = (grep defined, @list)[0]; # first defined value in @list
Bear in mind that first may be much more efficient if the list is
large and/or the test is expensive:
$ cat foo.pl
use List::Util qw( first );
sub even {
my ( $where, $x ) = @_;
print "$where is checking $x\n";
return $x % 2 == 0;
}
my @in = ( 1, 2, 3, 4, 5 );
my $first_even = first { even( 'first', $_ ) } @in;
my $grep_even = ( grep { even( 'grep', $_ ) } @in )[0];
print "$first_even, $grep_even\n";
$ perl foo.pl
first is checking 1
first is checking 2
grep is checking 1
grep is checking 2
grep is checking 3
grep is checking 4
grep is checking 5
2, 2
I'm not suggesting that necessarily applies in this case - just a
general observation.
--
Andy Armstrong, Hexten
Re: How to debug bizarre memory corruption in mod_perl
On Wed, 9 Jul 2008 13:23:13 +0100
Andy Armstrong <[EMAIL PROTECTED]> wrote:
> On 9 Jul 2008, at 09:09, David Kaufman wrote:
> > # from the POD doco at
> > #
> > http://search.cpan.org/~gbarr/Scalar-List-Utils-1.19/lib/List/Util.pm#DESCRIPTION
> >
> > $foo = first { defined($_) } @list; # first defined value in @list
> >
> >
> > Who needs to install a CPAN module to do that? I personally would
> > have
> > written it as:
> >
> > $foo = (grep defined, @list)[0]; # first defined value in @list
>
>
> Bear in mind that first may be much more efficient if the list is
> large and/or the test is expensive
Agree on the efficiency. I worked with a company on a large
mod_perl application (300K+ lines of code) that used a particular
function VERY often, that took an array and returned all the
unique values.
It did this with the common Perl Cookbook way of putting everything
into a hash and returning the keys(). I switched that function
to using List::MoreUtils::uniq() and they saw almost a 10% speed
improvement to their entire application. Granted this was a special
case because of how often that particular function was used (which
was a bad design decision in the first place), but sometimes
using these types of modules is useful/necessary even if the
normal "Perl core" way of doing it is relatively simple.
For example, if we use the technique above vs first() you get:
100 items in the list:
Rate Grep first
Grep 85470/s-- -74%
first 331126/s 287%--
5000 items in the list:
Rate Grep first
Grep1743/s-- -99%
first 125000/s 7073%--
So even for relatively small lists it can be a huge performance
win.
---
Frank Wiles, Revolution Systems, LLC.
Personal : [EMAIL PROTECTED] http://www.wiles.org
Work : [EMAIL PROTECTED] http://www.revsys.com
Re: How to debug bizarre memory corruption in mod_perl
Well, thank you for your advice, gentlemen. It got me on the right track, and I did manage to figure out the issue today. You may or may not be surprised to find it was this: my $foo = "bar" if $baz; Apparently one of the previous programmers left about 100 such constructs littered about the Perl and Mason code. Fixing them cleared up all the bizarre issues. We had Perl::Critic for the Perl side of the app, and I've done some hacks to criticize the Mason code, but someone had disabled the ProhibitConditionalDeclarations policy. *facepalm* It's interesting that this didn't manifest under Perl 5.8. Not sure if it's worth trying to develop a test case, since my...if falls into the "never do this" category anyway. -- Stephen Clouse <[EMAIL PROTECTED]>
Re: Apache2::Request undefined symbol
Mark Hedges wrote: That's a normal thing. All installed modules put their config into mods-available. Then you use `a2enmod` to manage those symlinks and turn them on or off in mods-enabled. +1 I believe all Debian-based distributions have done this with Apache 2 for a few years now. It scared me quite nicely when I upgraded the first time they did this (on Debian "unstable", I believe) and found none of my modules were enabled anymore and my Apache configuration had been overwritten. :) Colin
RE: Apache2::Request undefined symbol
Yes, it's not obvious that you need to enable the module after installing it. Why doesn't the installer enable it automatically? -Original Message- From: Colin Wetherbee [mailto:[EMAIL PROTECTED] Sent: Thursday, 10 July 2008 12:52 PM To: Mark Hedges Cc: Paul Cameron; modperl@perl.apache.org Subject: Re: Apache2::Request undefined symbol Mark Hedges wrote: > That's a normal thing. All installed modules put their config into > mods-available. Then you use `a2enmod` to manage those symlinks and > turn them on or off in mods-enabled. +1 I believe all Debian-based distributions have done this with Apache 2 for a few years now. It scared me quite nicely when I upgraded the first time they did this (on Debian "unstable", I believe) and found none of my modules were enabled anymore and my Apache configuration had been overwritten. :) Colin
Re: How to debug bizarre memory corruption in mod_perl
On Wed, Jul 9, 2008 at 9:23 PM, Stephen Clouse <[EMAIL PROTECTED]> wrote: > We had Perl::Critic for the Perl side of the app, and I've done some hacks > to criticize the Mason code, but someone had disabled the > ProhibitConditionalDeclarations policy. I'm thrilled to hear that this ultimately was valuable because I suggested that policy. It would be great if you'd share your hack to make Critic work on Mason code somewhere. > It's interesting that this didn't manifest under Perl 5.8. And a little scary, since some people definitely have this bug and the warning for it in 5.10 was removed before release. - Perrin
Re: How to debug bizarre memory corruption in mod_perl
On Wed, Jul 9, 2008 at 11:05 PM, Perrin Harkins <[EMAIL PROTECTED]> wrote:
> I'm thrilled to hear that this ultimately was valuable because I
> suggested that policy. It would be great if you'd share your hack to
> make Critic work on Mason code somewhere.
Oh, yeah, sure. Here's my test script. It just takes the compiled Mason
object and runs it through Critic. Also, the last half is shamelessly
cribbed from Test::Perl::Critic.
This is less useful than it seems because Mason's object code output doesn't
pass Perl::Critic itself :) But it's probably a start for someone to
improve upon -- maybe modulize it and throw it on CPAN. Or maybe improve
Mason's code output so it does pass a critique.
#!/usr/bin/perl
use strict;
use warnings;
no warnings qw(once);
use File::Find ();
use HTML::Mason::Compiler::ToObject ();
use Perl::Critic ();
use Test::More;
my $critic = Perl::Critic->new(
-profile => 't/perlcriticrc',
);
local *HTML::Mason::Commands::r = \"0";
my @files;
my $wanted = sub {
return if -d $_;
return if $_ eq 'favicon.ico';
return if $File::Find::name =~ /\.svn|images|javascript|css/;
push @files, $File::Find::name;
};
File::Find::find($wanted, 'htdocs');
plan tests => scalar(@files);
foreach my $file (@files) {
my(@vios);
eval {
my $src;
{ local $/ = undef;
open(my $foo, $file) or die "Couldn't open $file: $!";
$src = <$foo>;
}
my $compiler = HTML::Mason::Compiler::ToObject->new;
my $object_code = $compiler->compile(
comp_source => \$src,
name => $file,
comp_path => 'htdocs',
);
@vios = $critic->critique($object_code, -severity => 1);
};
if ($@) {
fail($file);
diag("\n");
diag(qq{Perl::Critic had errors in "$file":});
diag([EMAIL PROTECTED]);
} elsif (@vios) {
fail($file);
diag("\n");
diag(qq{Perl::Critic found these violations in "$file":});
my $verbose = $critic->config->verbose();
Perl::Critic::Violation::set_format( $verbose );
foreach my $vio (@vios) { diag("$vio") }
} else {
pass($file);
}
}
--
Stephen Clouse <[EMAIL PROTECTED]>
some flawed benchmarks
A couple of months ago i was going through slides from gozers "From
CGI to mod_perl 2.0, Fast!" talk, which has some benchmarks comparing
CGI, perlrun and registry to each other. At which point i realized
that i've never really known how much faster using straight handlers
is than using one of the CGI emulation layers. I also didn't have any
idea how much faster SetHandler modperl was vs SetHandler perl-script.
So i decided to see what i could figure out. I took gozers CGI from
the slides (slightly modified) and ran it through the paces on my
laptop, then converted the script to run as a straight handler.
here's the CGI version:
#!/usr/bin/perl
print qq[Content-Type: text/html\r\n\r\n];
print(qq[
Hello Worlds
GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}
MOD_PERL: $ENV{MOD_PERL}
]);
Here's the Handler version
package Kabob::HelloWorld;
use strict;
use warnings;
use Apache2::RequestRec ();
use Apache2::Const -compile =>qw(:common);
sub handler {
my $r = shift;
$r->content_type('text/html');
$r->print(qq[
Hello Worlds
GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}
MOD_PERL: $ENV{MOD_PERL}
]);
return Apache2::Const::OK;
}
1;
and here's the conf (these tests were all running through a light
mod_proxy front end too)
ProxyPass http://localhost:8080/cgi/
ProxyPassReverse http://localhost:8080/cgi/
ScriptAlias /cgi/ /www/p/
Alias /perlrun/ /www/p/
ProxyPass http://localhost:8080/perlrun/
ProxyPassReverse http://localhost:8080/perlrun/
SetHandler perl-script
PerlHandler ModPerl::PerlRun
Options +ExecCGI
PerlSendHeader On
Alias /registry/ /www/p/
ProxyPass http://localhost:8080/registry/
ProxyPassReverse http://localhost:8080/registry/
SetHandler perl-script
PerlHandler ModPerl::Registry
Options +ExecCGI
PerlSendHeader On
ProxyPass http://localhost:8080/perlscript/
ProxyPassReverse http://localhost:8080/perlscript/
SetHandler perl-script
PerlResponseHandler Kabob::HelloWorld
ProxyPass http://localhost:8080/modperl/
ProxyPassReverse http://localhost:8080/modperl/
SetHandler modperl
PerlResponseHandler Kabob::HelloWorld
and here's the results (which are no doubt flawed for a number of reasons)
running: ab -n 1 [url]
CGI
Requests per second:217.80 [#/sec] (mean)
Time per request: 4.591 [ms] (mean)
Transfer rate: 53.17 [Kbytes/sec] received
PerlRun
Requests per second:482.49 [#/sec] (mean)
Time per request: 2.073 [ms] (mean)
Transfer rate: 114.49 [Kbytes/sec] received
Registry
Requests per second:693.33 [#/sec] (mean)
Time per request: 1.442 [ms] (mean)
Transfer rate: 164.53 [Kbytes/sec] received
SetHandler perl-script
Requests per second:772.12 [#/sec] (mean)
Time per request: 1.295 [ms] (mean)
Transfer rate: 189.94 [Kbytes/sec] received
SetHandler modperl
Requests per second:1048.66 [#/sec] (mean)
Time per request: 0.954 [ms] (mean)
Transfer rate: 250.84 [Kbytes/sec] received
I'm not sure how well you can really compare the CGI emulation numbers
to the PerlHandler numbers, but personally i think the 30%ish
improvement from perl-script to modperl is pretty amazing. I wouldn't
have imagined it would have been that high.
Adam
