Re: How to debug bizarre memory corruption in mod_perl
"Stephen Clouse" <[EMAIL PROTECTED]> wrote: > Under [mod_perl 2 and perl 5.10 on Fedora 9] I am getting some of the > most bizarre and insidious perl core errors I've ever seen in my 15 > years of using perl. > > Attempt to free unreferenced scalar: SV 0xbd266be4, Perl interpreter: > 0xba01c410 at /usr/lib/perl5/vendor_perl/5.10.0/HTML/Mason/Component.pm > line > 147. > panic: attempt to copy value 20 to a freed scalar bd49dffc > Use of freed value in iteration > panic: free from wrong pool > > Not to mention countless application errors involving dereferencing. Dereferencing demons, eh? Your error rang a bell with me. But it wasn't until I saw List/Util/Util.so in your follow-up post that I remembered, List::Util did that same scary crashing thing to me a while back while I was debugging krang (which is mod_perl 1.3x-based). It boiled down to the (IMO) senseless and unnecessary use of weak referrences somewhere deep in the dark bowels of the Scalar-List-Utils package. I'd read somewhere that weak references were not all that stable in certain recent versions of perl, so when I saw all the weakening being done in there (and all the XS bugs the module has had to fix, in its change log: http://search.cpan.org/src/GBARR/Scalar-List-Utils-1.19/Changes At the time, upgrading the module to the latest release didn't stop my big scary memory reference errors. So I read through the code that used the module and was astonished at why the module was being used: the *one* line of code in the entire module that I was troubleshooting that *depended* on List::Util was a single, simple call to its exported method: first() What does first() do? It invokes kilobytes of fast, compiled XS code to ...huh? Return to me the first element in a list I pass to it that returns true for a code block I pass to it? Yup. Like this: # from the POD doco at # http://search.cpan.org/~gbarr/Scalar-List-Utils-1.19/lib/List/Util.pm#DESCRIPTION $foo = first { defined($_) } @list; # first defined value in @list Who needs to install a CPAN module to do that? I personally would have written it as: $foo = (grep defined, @list)[0]; # first defined value in @list But thats kind of non-obvious to the programmer who finds it later (if they dont read the comment, that is). You could also simply replace "first" with "grep" (and a pair of parens) and get the same effect: ($foo) = grep { defined($_) } @list; # first defined value in @list (at the cost of one extra byte and some wasted cpu cycles if the compiler doesn't notice that the lvalue is a list of length one, and lets grep process the whole list anyway, even after it finds and assigns a defined element in there). IMO, even this would be better than loading List::Utils and calling first(): my $foo; for (@list) { next unless defined($_); $foo = $_; last; } So I fixed my bug by writing a line or two of perl to in order to remove one call to the List::Util module's first() function. > My problem is that I don't even know where or how to start tracing or > debugging such an issue to determine precisely what is scribbling on > perl's > memory. Starting from your error message (line 147 of HTML::Mason::Component) it appears HTML::Mason is calling a "dynamic method" method (one named in a variable). No help there. Okay, what else is HTML::Mason::Component using? What libraries has it loaded? Is List::Utils being used? This looked suspicious to me: use HTML::Mason::Tools qw(absolute_comp_path can_weaken); Just because you "can_weaken" a reference, does that mean you should? Sure enough HTML::Mason::Tools loads Scalar::Util apparently to do some reference weakening. I'd check my Mason component for use of the functions that the Scalar-List-Utils modules provide: List::Util first max maxstr min minstr reduce shuffle sum Scalar::Util blessed dualvar isweak readonly refaddr reftype tainted weaken isvstring looks_like_number set_prototype And see if rewriting these in pure perl helps. Also, if it does, but would require changing code that you'd rather not monkey with, it looks like there is a pure-perl version of the module that can be installed by passing a -pm option to Makefile.PL: perl Makefile.PL -pm make make test make install Anyway, I hope somehow, some of this helps! -dave
Re: How to debug bizarre memory corruption in mod_perl
On 9 Jul 2008, at 09:09, David Kaufman wrote: # from the POD doco at # http://search.cpan.org/~gbarr/Scalar-List-Utils-1.19/lib/List/Util.pm#DESCRIPTION $foo = first { defined($_) } @list; # first defined value in @list Who needs to install a CPAN module to do that? I personally would have written it as: $foo = (grep defined, @list)[0]; # first defined value in @list Bear in mind that first may be much more efficient if the list is large and/or the test is expensive: $ cat foo.pl use List::Util qw( first ); sub even { my ( $where, $x ) = @_; print "$where is checking $x\n"; return $x % 2 == 0; } my @in = ( 1, 2, 3, 4, 5 ); my $first_even = first { even( 'first', $_ ) } @in; my $grep_even = ( grep { even( 'grep', $_ ) } @in )[0]; print "$first_even, $grep_even\n"; $ perl foo.pl first is checking 1 first is checking 2 grep is checking 1 grep is checking 2 grep is checking 3 grep is checking 4 grep is checking 5 2, 2 I'm not suggesting that necessarily applies in this case - just a general observation. -- Andy Armstrong, Hexten
Re: How to debug bizarre memory corruption in mod_perl
On Wed, 9 Jul 2008 13:23:13 +0100 Andy Armstrong <[EMAIL PROTECTED]> wrote: > On 9 Jul 2008, at 09:09, David Kaufman wrote: > > # from the POD doco at > > # > > http://search.cpan.org/~gbarr/Scalar-List-Utils-1.19/lib/List/Util.pm#DESCRIPTION > > > > $foo = first { defined($_) } @list; # first defined value in @list > > > > > > Who needs to install a CPAN module to do that? I personally would > > have > > written it as: > > > > $foo = (grep defined, @list)[0]; # first defined value in @list > > > Bear in mind that first may be much more efficient if the list is > large and/or the test is expensive Agree on the efficiency. I worked with a company on a large mod_perl application (300K+ lines of code) that used a particular function VERY often, that took an array and returned all the unique values. It did this with the common Perl Cookbook way of putting everything into a hash and returning the keys(). I switched that function to using List::MoreUtils::uniq() and they saw almost a 10% speed improvement to their entire application. Granted this was a special case because of how often that particular function was used (which was a bad design decision in the first place), but sometimes using these types of modules is useful/necessary even if the normal "Perl core" way of doing it is relatively simple. For example, if we use the technique above vs first() you get: 100 items in the list: Rate Grep first Grep 85470/s-- -74% first 331126/s 287%-- 5000 items in the list: Rate Grep first Grep1743/s-- -99% first 125000/s 7073%-- So even for relatively small lists it can be a huge performance win. --- Frank Wiles, Revolution Systems, LLC. Personal : [EMAIL PROTECTED] http://www.wiles.org Work : [EMAIL PROTECTED] http://www.revsys.com
Re: How to debug bizarre memory corruption in mod_perl
Well, thank you for your advice, gentlemen. It got me on the right track, and I did manage to figure out the issue today. You may or may not be surprised to find it was this: my $foo = "bar" if $baz; Apparently one of the previous programmers left about 100 such constructs littered about the Perl and Mason code. Fixing them cleared up all the bizarre issues. We had Perl::Critic for the Perl side of the app, and I've done some hacks to criticize the Mason code, but someone had disabled the ProhibitConditionalDeclarations policy. *facepalm* It's interesting that this didn't manifest under Perl 5.8. Not sure if it's worth trying to develop a test case, since my...if falls into the "never do this" category anyway. -- Stephen Clouse <[EMAIL PROTECTED]>
Re: Apache2::Request undefined symbol
Mark Hedges wrote: That's a normal thing. All installed modules put their config into mods-available. Then you use `a2enmod` to manage those symlinks and turn them on or off in mods-enabled. +1 I believe all Debian-based distributions have done this with Apache 2 for a few years now. It scared me quite nicely when I upgraded the first time they did this (on Debian "unstable", I believe) and found none of my modules were enabled anymore and my Apache configuration had been overwritten. :) Colin
RE: Apache2::Request undefined symbol
Yes, it's not obvious that you need to enable the module after installing it. Why doesn't the installer enable it automatically? -Original Message- From: Colin Wetherbee [mailto:[EMAIL PROTECTED] Sent: Thursday, 10 July 2008 12:52 PM To: Mark Hedges Cc: Paul Cameron; modperl@perl.apache.org Subject: Re: Apache2::Request undefined symbol Mark Hedges wrote: > That's a normal thing. All installed modules put their config into > mods-available. Then you use `a2enmod` to manage those symlinks and > turn them on or off in mods-enabled. +1 I believe all Debian-based distributions have done this with Apache 2 for a few years now. It scared me quite nicely when I upgraded the first time they did this (on Debian "unstable", I believe) and found none of my modules were enabled anymore and my Apache configuration had been overwritten. :) Colin
Re: How to debug bizarre memory corruption in mod_perl
On Wed, Jul 9, 2008 at 9:23 PM, Stephen Clouse <[EMAIL PROTECTED]> wrote: > We had Perl::Critic for the Perl side of the app, and I've done some hacks > to criticize the Mason code, but someone had disabled the > ProhibitConditionalDeclarations policy. I'm thrilled to hear that this ultimately was valuable because I suggested that policy. It would be great if you'd share your hack to make Critic work on Mason code somewhere. > It's interesting that this didn't manifest under Perl 5.8. And a little scary, since some people definitely have this bug and the warning for it in 5.10 was removed before release. - Perrin
Re: How to debug bizarre memory corruption in mod_perl
On Wed, Jul 9, 2008 at 11:05 PM, Perrin Harkins <[EMAIL PROTECTED]> wrote: > I'm thrilled to hear that this ultimately was valuable because I > suggested that policy. It would be great if you'd share your hack to > make Critic work on Mason code somewhere. Oh, yeah, sure. Here's my test script. It just takes the compiled Mason object and runs it through Critic. Also, the last half is shamelessly cribbed from Test::Perl::Critic. This is less useful than it seems because Mason's object code output doesn't pass Perl::Critic itself :) But it's probably a start for someone to improve upon -- maybe modulize it and throw it on CPAN. Or maybe improve Mason's code output so it does pass a critique. #!/usr/bin/perl use strict; use warnings; no warnings qw(once); use File::Find (); use HTML::Mason::Compiler::ToObject (); use Perl::Critic (); use Test::More; my $critic = Perl::Critic->new( -profile => 't/perlcriticrc', ); local *HTML::Mason::Commands::r = \"0"; my @files; my $wanted = sub { return if -d $_; return if $_ eq 'favicon.ico'; return if $File::Find::name =~ /\.svn|images|javascript|css/; push @files, $File::Find::name; }; File::Find::find($wanted, 'htdocs'); plan tests => scalar(@files); foreach my $file (@files) { my(@vios); eval { my $src; { local $/ = undef; open(my $foo, $file) or die "Couldn't open $file: $!"; $src = <$foo>; } my $compiler = HTML::Mason::Compiler::ToObject->new; my $object_code = $compiler->compile( comp_source => \$src, name => $file, comp_path => 'htdocs', ); @vios = $critic->critique($object_code, -severity => 1); }; if ($@) { fail($file); diag("\n"); diag(qq{Perl::Critic had errors in "$file":}); diag([EMAIL PROTECTED]); } elsif (@vios) { fail($file); diag("\n"); diag(qq{Perl::Critic found these violations in "$file":}); my $verbose = $critic->config->verbose(); Perl::Critic::Violation::set_format( $verbose ); foreach my $vio (@vios) { diag("$vio") } } else { pass($file); } } -- Stephen Clouse <[EMAIL PROTECTED]>
some flawed benchmarks
A couple of months ago i was going through slides from gozers "From CGI to mod_perl 2.0, Fast!" talk, which has some benchmarks comparing CGI, perlrun and registry to each other. At which point i realized that i've never really known how much faster using straight handlers is than using one of the CGI emulation layers. I also didn't have any idea how much faster SetHandler modperl was vs SetHandler perl-script. So i decided to see what i could figure out. I took gozers CGI from the slides (slightly modified) and ran it through the paces on my laptop, then converted the script to run as a straight handler. here's the CGI version: #!/usr/bin/perl print qq[Content-Type: text/html\r\n\r\n]; print(qq[ Hello Worlds GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE} MOD_PERL: $ENV{MOD_PERL} ]); Here's the Handler version package Kabob::HelloWorld; use strict; use warnings; use Apache2::RequestRec (); use Apache2::Const -compile =>qw(:common); sub handler { my $r = shift; $r->content_type('text/html'); $r->print(qq[ Hello Worlds GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE} MOD_PERL: $ENV{MOD_PERL} ]); return Apache2::Const::OK; } 1; and here's the conf (these tests were all running through a light mod_proxy front end too) ProxyPass http://localhost:8080/cgi/ ProxyPassReverse http://localhost:8080/cgi/ ScriptAlias /cgi/ /www/p/ Alias /perlrun/ /www/p/ ProxyPass http://localhost:8080/perlrun/ ProxyPassReverse http://localhost:8080/perlrun/ SetHandler perl-script PerlHandler ModPerl::PerlRun Options +ExecCGI PerlSendHeader On Alias /registry/ /www/p/ ProxyPass http://localhost:8080/registry/ ProxyPassReverse http://localhost:8080/registry/ SetHandler perl-script PerlHandler ModPerl::Registry Options +ExecCGI PerlSendHeader On ProxyPass http://localhost:8080/perlscript/ ProxyPassReverse http://localhost:8080/perlscript/ SetHandler perl-script PerlResponseHandler Kabob::HelloWorld ProxyPass http://localhost:8080/modperl/ ProxyPassReverse http://localhost:8080/modperl/ SetHandler modperl PerlResponseHandler Kabob::HelloWorld and here's the results (which are no doubt flawed for a number of reasons) running: ab -n 1 [url] CGI Requests per second:217.80 [#/sec] (mean) Time per request: 4.591 [ms] (mean) Transfer rate: 53.17 [Kbytes/sec] received PerlRun Requests per second:482.49 [#/sec] (mean) Time per request: 2.073 [ms] (mean) Transfer rate: 114.49 [Kbytes/sec] received Registry Requests per second:693.33 [#/sec] (mean) Time per request: 1.442 [ms] (mean) Transfer rate: 164.53 [Kbytes/sec] received SetHandler perl-script Requests per second:772.12 [#/sec] (mean) Time per request: 1.295 [ms] (mean) Transfer rate: 189.94 [Kbytes/sec] received SetHandler modperl Requests per second:1048.66 [#/sec] (mean) Time per request: 0.954 [ms] (mean) Transfer rate: 250.84 [Kbytes/sec] received I'm not sure how well you can really compare the CGI emulation numbers to the PerlHandler numbers, but personally i think the 30%ish improvement from perl-script to modperl is pretty amazing. I wouldn't have imagined it would have been that high. Adam