Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
According to our install documentation, these are the apache2 packages that we install from apt on debian jessie: apache2 apache2-mpm-prefork libapache2-request-perl libapache2-mod-perl2 libapache2-mod-perl2-dev apache2-threaded-dev On Thu, Sep 17, 2015 at 2:13 PM, Michael Schout wrote: > On 9/17/15 8:13 AM, Vincent Veyron wrote: > > > Sep 17 15:08:31 eeepc kernel: [ 7737.194398] traps: > /usr/sbin/apach[8769] general protection ip:7f1199d45d70 sp:7ffe7e3dc410 > error:0 in PPAddr.so[7f1199d45000+2000] > > I saw these first as well on debian jessie. I fixed it by switching > from mpm-event to mpm-prefork. > > I will say that I find debian's packaging to makes it pretty difficult > with Apache::Test. Apache::Test fails to generate a working config > because it does not source the Include files in > /etc/apache2/mods-enabled (among other things). I much prefer to just > build apache from scratch, which also eliminates the possibility that a > vendor applied patch is causing any problems. > > Here is how I produced the panic on jessie, using nothing but vendor > packages if you *really* do want to reproduce this: > > In fresh debian 8/jessie do this (I used a vagrant box): > > sudo apt-get update > > sudo apt-get install apache2-mpm-prefork libapache2-request-perl \ > libapache2-mod-perl2 libapache2-mod-apreq2 \ > apache2-dev git libtrycatch-perl > > git clone git://github.com/mschout/modperl-trycatch-bug > > cd modperl-trycatch-bug > > # now we have to jump through several hoops to make Apache::Test work > > # make symlinks that apache2.conf needs *sigh* > ln -s /etc/apache2/mods-enabled t/mods-enabled > ln -s /etc/apache2/conf-enabled t/conf-enabled > > # disable mpm-event and use mpm-prefork instead > cd /etc/apache2/mods-enabled/ > sudo rm -f mpm_event.* > sudo ln -s ../mods-available/mpm_prefork.conf mpm_prefork.conf > sudo ln -s ../mods-available/mpm_prefork.load mpm_prefork.load > cd - > > # change IncludeOptional to Include > sudo sed -i 's/IncludeOptional/Include/' /etc/apache2/apache2.conf > > # need to source apache2 envvars > source /etc/apache2/envvars > > # now actually run the test > perl Makefile.PL -apxs /usr/bin/apxs2 \ > -httpd_conf /etc/apache2/apache2.conf > > make test > > ... which fails ... > > t/logs/error_log contains: > > [Thu Sep 17 20:08:09.333854 2015] [perl:error] [pid 10319] [client > 127.0.0.1:60632] panic: attempt to copy freed scalar 7f4e0971ec80 to > 7f4e0971eb30 at > /home/vagrant/modperl-trycatch-bug/blib/lib/Apache/TryCatch.pm line 20.\n > > > Regards, > Michael Schout > -- John Dunlap *CTO | Lariat * *Direct:* *j...@lariat.co * *Customer Service:* 877.268.6667 supp...@lariat.co
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
On 9/17/15 8:13 AM, Vincent Veyron wrote: > Sep 17 15:08:31 eeepc kernel: [ 7737.194398] traps: /usr/sbin/apach[8769] > general protection ip:7f1199d45d70 sp:7ffe7e3dc410 error:0 in > PPAddr.so[7f1199d45000+2000] I saw these first as well on debian jessie. I fixed it by switching from mpm-event to mpm-prefork. I will say that I find debian's packaging to makes it pretty difficult with Apache::Test. Apache::Test fails to generate a working config because it does not source the Include files in /etc/apache2/mods-enabled (among other things). I much prefer to just build apache from scratch, which also eliminates the possibility that a vendor applied patch is causing any problems. Here is how I produced the panic on jessie, using nothing but vendor packages if you *really* do want to reproduce this: In fresh debian 8/jessie do this (I used a vagrant box): sudo apt-get update sudo apt-get install apache2-mpm-prefork libapache2-request-perl \ libapache2-mod-perl2 libapache2-mod-apreq2 \ apache2-dev git libtrycatch-perl git clone git://github.com/mschout/modperl-trycatch-bug cd modperl-trycatch-bug # now we have to jump through several hoops to make Apache::Test work # make symlinks that apache2.conf needs *sigh* ln -s /etc/apache2/mods-enabled t/mods-enabled ln -s /etc/apache2/conf-enabled t/conf-enabled # disable mpm-event and use mpm-prefork instead cd /etc/apache2/mods-enabled/ sudo rm -f mpm_event.* sudo ln -s ../mods-available/mpm_prefork.conf mpm_prefork.conf sudo ln -s ../mods-available/mpm_prefork.load mpm_prefork.load cd - # change IncludeOptional to Include sudo sed -i 's/IncludeOptional/Include/' /etc/apache2/apache2.conf # need to source apache2 envvars source /etc/apache2/envvars # now actually run the test perl Makefile.PL -apxs /usr/bin/apxs2 \ -httpd_conf /etc/apache2/apache2.conf make test ... which fails ... t/logs/error_log contains: [Thu Sep 17 20:08:09.333854 2015] [perl:error] [pid 10319] [client 127.0.0.1:60632] panic: attempt to copy freed scalar 7f4e0971ec80 to 7f4e0971eb30 at /home/vagrant/modperl-trycatch-bug/blib/lib/Apache/TryCatch.pm line 20.\n Regards, Michael Schout
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
It's not one of my direct dependencies and I don't believe that it is a transitive dependency either because it isn't installed on my test server. On Thu, Sep 17, 2015 at 1:26 PM, Michael Schout wrote: > On 9/17/15 9:00 AM, John Dunlap wrote: > > I'm not using the TryCatch module in my application. I use eval blocks > > for all of my transaction management. For me to be suffering from the > > same ailment, the bug would have to manifest itself in other ways. > > I don't think TryCatch *itself* is the problem. I was seeing > segmentation faults from other sections of code that did not use > TryCatch that the op.c change apparently also causes. > > There is some other underlying problem that is causing this. You can > put other chunks of code inside the try { } block and the panic does not > happen. For some reason $cgi->param() inside the try *does* trigger it, > but it also needs to be called from a subroutine (e.g.: moving the try > block inside the handler() sub does not cause the panic. This all sort > of makes sense based on what the change to op.c does to the optree that > causes the bug (skips over "return" ops that do nothing basically) > > In addition I had other segfaults that DO NOT issue the "panic" warning. > But reversing the optimization to op.c in perl fixes that for me also. > > I haven't had time to really try to get into the modperl code enough to > know what is going on, but the backtrace from panic is coming from > copying scalars that have flags or something to that effect when leaving > a subroutine. I'm pretty sure modperl messes with scalar flags and/or > scalar magic due to previous experience with the mod_perl codebase a > long time ago so that might explain why this ONLY happens under modperl > and not under vanilla perl. > > Also, TryCatch is implemented using Devel::Declare, which is admittedly > sneaky because it screws with the perl optree during compilation. The > part that was changed in op.c that causes the problems this thread is > about essentially "skip over" a couple of unnecessary ops as an > optimization. > > So this is a bit long winded, but I kind of doubt TryCatch is the > problem. TryCatch+CGI->param called from a soubroutine just happens to > be one of the shortest, sure fire ways to trigger the panic that I have > found when I needed to make a test case to demonstrate the bug. > > I suspect Devel::Declare was getting away with something that it > shouldn't be able to do, or, something in modperl at this time. > > Is Devel::Declare a dependency anywhere in your codebase? E.g.: if you > have Devel/Declare.pm in your perl library path, will you code still run > if you remove Devel/Declare.pm? If so then that either means the > problem is probably in modperl itself, or, you are seeing something > completely different. > > Regards, > Michael Schout > -- John Dunlap *CTO | Lariat * *Direct:* *j...@lariat.co * *Customer Service:* 877.268.6667 supp...@lariat.co
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
On 9/17/15 9:00 AM, John Dunlap wrote: > I'm not using the TryCatch module in my application. I use eval blocks > for all of my transaction management. For me to be suffering from the > same ailment, the bug would have to manifest itself in other ways. I don't think TryCatch *itself* is the problem. I was seeing segmentation faults from other sections of code that did not use TryCatch that the op.c change apparently also causes. There is some other underlying problem that is causing this. You can put other chunks of code inside the try { } block and the panic does not happen. For some reason $cgi->param() inside the try *does* trigger it, but it also needs to be called from a subroutine (e.g.: moving the try block inside the handler() sub does not cause the panic. This all sort of makes sense based on what the change to op.c does to the optree that causes the bug (skips over "return" ops that do nothing basically) In addition I had other segfaults that DO NOT issue the "panic" warning. But reversing the optimization to op.c in perl fixes that for me also. I haven't had time to really try to get into the modperl code enough to know what is going on, but the backtrace from panic is coming from copying scalars that have flags or something to that effect when leaving a subroutine. I'm pretty sure modperl messes with scalar flags and/or scalar magic due to previous experience with the mod_perl codebase a long time ago so that might explain why this ONLY happens under modperl and not under vanilla perl. Also, TryCatch is implemented using Devel::Declare, which is admittedly sneaky because it screws with the perl optree during compilation. The part that was changed in op.c that causes the problems this thread is about essentially "skip over" a couple of unnecessary ops as an optimization. So this is a bit long winded, but I kind of doubt TryCatch is the problem. TryCatch+CGI->param called from a soubroutine just happens to be one of the shortest, sure fire ways to trigger the panic that I have found when I needed to make a test case to demonstrate the bug. I suspect Devel::Declare was getting away with something that it shouldn't be able to do, or, something in modperl at this time. Is Devel::Declare a dependency anywhere in your codebase? E.g.: if you have Devel/Declare.pm in your perl library path, will you code still run if you remove Devel/Declare.pm? If so then that either means the problem is probably in modperl itself, or, you are seeing something completely different. Regards, Michael Schout
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
I'm not using the TryCatch module in my application. I use eval blocks for all of my transaction management. For me to be suffering from the same ailment, the bug would have to manifest itself in other ways. On Sep 17, 2015 5:01 AM, "Vincent Veyron" wrote: > On Wed, 16 Sep 2015 12:36:07 -0500 > Michael Schout wrote: > > > >The bug is definitely there on jessie > > I'm seeing it too, now : > > [Thu Sep 17 10:20:46.609613 2015] [perl:error] [pid 1162:tid > 140075585201920] [client 192.168.1.11:44374] panic: attempt to copy freed > scalar 7f65d1541178 to 7f65d0e6fc30 at /home/lib/Apache/TryCatch.pm line > 20.\n > > Don't know if it's of any use, but my server has been behaving strangely : > > 1 - it won't start with your dependency; there is no error message in the > logs, no visible Apache process. Yet, I need to do a full 'service apache2 > stop' to complete step 2 > 2 - comment 'use TryCatch;' in your script and the server starts normally, > then uncomment the line (the change being picked up by Apache2::Reload) to > trigger the bug. > > > Sorry about the noise. > > -- > Bien à vous, Vincent Veyron > > https://legalcase.libremen.com/ > Legal case, contract and insurance claim management software >
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
Correction : I found the following in /var/log/syslog when attempting to start the server with the 'use TryCatch' dependency Sep 17 15:08:31 eeepc kernel: [ 7737.194398] traps: /usr/sbin/apach[8769] general protection ip:7f1199d45d70 sp:7ffe7e3dc410 error:0 in PPAddr.so[7f1199d45000+2000] Sep 17 15:08:31 eeepc apache2[8756]: Starting web server: apache2Segmentation fault Sep 17 15:08:31 eeepc apache2[8756]: Action 'start' failed. Sep 17 15:08:31 eeepc apache2[8756]: The Apache error log may have more information. Apache log is empty, though. -- Bien à vous, Vincent Veyron https://marica.fr/ Gestion des contentieux, des dossiers de sinistres assurance et des contrats pour le service juridique
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
On Wed, 16 Sep 2015 12:36:07 -0500 Michael Schout wrote: >The bug is definitely there on jessie I'm seeing it too, now : [Thu Sep 17 10:20:46.609613 2015] [perl:error] [pid 1162:tid 140075585201920] [client 192.168.1.11:44374] panic: attempt to copy freed scalar 7f65d1541178 to 7f65d0e6fc30 at /home/lib/Apache/TryCatch.pm line 20.\n Don't know if it's of any use, but my server has been behaving strangely : 1 - it won't start with your dependency; there is no error message in the logs, no visible Apache process. Yet, I need to do a full 'service apache2 stop' to complete step 2 2 - comment 'use TryCatch;' in your script and the server starts normally, then uncomment the line (the change being picked up by Apache2::Reload) to trigger the bug. Sorry about the noise. -- Bien à vous, Vincent Veyron https://legalcase.libremen.com/ Legal case, contract and insurance claim management software
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
Here is a backtrace from the call to Perl_croak() that is triggered, from perl 5.19.7, mod_perl 2.0.9 both built with debugging enabled. > * thread #1: tid = 0x841d66, 0x000102b058af > mod_perl.so`Perl_croak(pat=0x000102c48681) + 95 at util.c:1581, queue = > 'com.apple.main-thread', stop reason = breakpoint 1.1 > * frame #0: 0x000102b058af > mod_perl.so`Perl_croak(pat=0x000102c48681) + 95 at util.c:1581 > frame #1: 0x000102b4d55c > mod_perl.so`Perl_sv_setsv_flags(dstr=, sstr=, > flags=1536) + 9708 at sv.c:4062 > frame #2: 0x000102b47824 > mod_perl.so`Perl_sv_mortalcopy_flags(oldstr=0x0001010c76b8, > flags=) + 452 at sv.c:8643 > frame #3: 0x000102b9ac1a > mod_perl.so`S_adjust_stack_on_leave(newsp=0x000100f96c00, > sp=0x000100f96c08, mark=0x000100f96c08, gimme=, > flags=, lvalue=false) + 250 at pp_ctl.c:2031 > frame #4: 0x000102ba87b1 mod_perl.so`Perl_pp_leavetry + 417 at > pp_ctl.c:4397 > frame #5: 0x000102b052a1 mod_perl.so`Perl_runops_debug + 305 at > dump.c:2270 > frame #6: 0x000102a86cbc mod_perl.so`Perl_call_sv(sv=, > flags=) + 1436 at perl.c:2747 > frame #7: 0x000102a3c45f > mod_perl.so`modperl_callback(handler=, p=, > r=0x0001008b00a0, s=, args=) + 607 at > modperl_callback.c:100 > frame #8: 0x000102a3d7bd > mod_perl.so`modperl_callback_run_handlers(idx=6, type=3, > r=0x0001008b00a0, c=, s=0x00010080e930, > pconf=, plog=, ptemp=, > run_mode=) + 829 at modperl_callback.c:236 > frame #9: 0x000102a3e506 > mod_perl.so`modperl_callback_per_dir(idx=, r=, > run_mode=) + 54 at modperl_callback.c:342 > frame #10: 0x000102a377e6 mod_perl.so`modperl_response_handler_cgi > [inlined] modperl_response_handler_run(r=0x0001008b00a0) + 34 at > mod_perl.c:1027 > frame #11: 0x000102a377c4 > mod_perl.so`modperl_response_handler_cgi(r=0x0001008b00a0) + 420 at > mod_perl.c:1098 > frame #12: 0x00012186 httpd`ap_invoke_handler + 310 > frame #13: 0x000100023fb5 httpd`ap_process_request + 85 > frame #14: 0x00010002161f httpd`ap_process_http_connection + 127 > frame #15: 0x0001ee26 httpd`ap_process_connection + 150 > frame #16: 0x00010002a87b httpd`child_main + 827 > frame #17: 0x000100029ea2 httpd`make_child + 114 > frame #18: 0x000100029489 httpd`ap_mpm_run + 1129 > frame #19: 0x0001771e httpd`main + 2142 > frame #20: 0x7fff95acc5c9 libdyld.dylib`start + 1 > frame #21: 0x7fff95acc5c9 libdyld.dylib`start + 1 Regards, Michael Schout
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
On 9/16/15 12:06 PM, Steve Hay wrote: > Thanks for the detective work, Michael. I've logged this on > rt.perl.org since it isn't yet clear whether the optimization has a > problem, or whether mod_perl (or maybe TryCatch / Devel::Declare) is > doing anything naughty: Thanks Steve. Wouldn't surprise me if it was any of the three. I will try to do more debugging as time allows. Regards, Michael Schout
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
On 9/16/15 12:26 PM, Vincent Veyron wrote: > Ha! It was not installed. I suppose that because your test module was also > named TryCatch, Probably. > Not sure what to do now? Rename the crash module as /home/lib/Apache/TryCatch.pm make sure the "package" line still reads "package Apache::TryCatch;" or, unless you are planning to debug perl internals, mod_perl internals, and/or TryCatch/Devel::Declare to try to trace down the root cause of this problem, just move on. The bug is definitely there on jessie as per my previous email showing how I reproduced it. If you do want to reproduce it, use the same apache2.conf additions from my last email. Regards, Michael Schout
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
On Wed, 16 Sep 2015 09:40:43 -0500 Michael Schout wrote: > On 9/16/15 9:22 AM, Vincent Veyron wrote: > > apt-get install apache2-mpm-prefork libapache2-request-perl > > libapache2-mod-perl2 libapache2-mod-apreq2 > > How did you install TryCatch (the only dependency my module uses)? > Ha! It was not installed. I suppose that because your test module was also named TryCatch, Apache used it instead in the dependency? I don't understand why the try function works, but anyhow... I installed TryCatch with cpan TryCatch And renamed your test module to crash.pm Now Apache won't start, here is the error : [Wed Sep 16 18:26:39.331849 2015] [perl:error] [pid 15699:tid 139626838587264] Can't locate TryCatch.pm in @INC (you may need to install the TryCatch module) (@INC contains: /home/lib @ServerRoot/../lib /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/x86_64-linux-gnu/perl5/5.20 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl /etc/apache2) at (eval 85) line 2.\n Even though 'locate TryCatch.pm' outputs : /usr/local/lib/x86_64-linux-gnu/perl/5.20.2/TryCatch.pm and the directory is in @INC. Also : perl -MTryCatch works from the command line Not sure what to do now? -- Bien à vous, Vincent Veyron https://marica.fr/ Gestion des contentieux, des dossiers de sinistres assurance et des contrats pour le service juridique
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
On 9/16/15 7:08 AM, Vincent Veyron wrote: > I put your test handler on a machine running Debian 8 (jessie), it works just > fine. I cannot reproduce this at all. I tried on Debian 8 jessie running in virtualbox. perl, modperl, apache all from debian packages. using prefork MPM. TryCatch installed from cpan or from debian packages, makes no difference at all. [Wed Sep 16 19:12:11.418852 2015] [perl:error] [pid 29001] [client 192.168.1.107:63580] panic: attempt to copy freed scalar 7f62b088a6d0 to 7f62b088a580 at /home/vagrant/modperl-trycatch-bug/lib/Apache/TryCatch.pm line 20.\n perl -v : This is perl 5, version 20, subversion 2 (v5.20.2) built for x86_64-linux-gnu-thread-multi dpkg -l *mod-perl* : ii libapache2-mod-perl2 2.0.9~1624218-2 amd64Integration of perl with the Apache2 web server apache2 -v: Server version: Apache/2.4.10 (Debian) Server built: Aug 28 2015 16:28:08 I did not use a startup.pl. Can't see that making any difference. I just added this at the end of apache2.conf: -- PerlSwitches -I/home/vagrant/modperl-trycatch-bug/lib PerlModule Apache::TryCatch SetHandler perl-script PerlResponseHandler Apache::TryCatch -- I also did it with a startup.pl in /home/lib like you say you have. apache2.conf: - PerlRequire /home/lib/startup.pl SetHandler perl-script PerlResponseHandler Apache::TryCatch - and /home/lib/Apache/TryCatch.pm: - package Apache::TryCatch; use Apache2::Const -compile => qw(OK); use TryCatch; use CGI; sub handler :method { my ($self, $r) = @_; $r->content_type('text/plain'); print $self->crash($r); return Apache2::Const::OK; } sub crash { my $cgi = CGI->new; try { $cgi->param(x => 'x') } return "OK"; } 1; - And if fails 100% of the time for me that way also when accessing /crashme: [Wed Sep 16 19:19:30.092140 2015] [perl:error] [pid 29154] [client 192.168.1.107:63594] panic: attempt to copy freed scalar 7f2bcf672758 to 7f2bcf672200 at /home/lib/Apache/TryCatch.pm line 20.\n Not sure what you have done differently, but the bug is definitely there on debian 8. Regards, Michael Schout
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
On 15 September 2015 at 23:15, Michael Schout wrote: > As reported in other thread, I've seen mod_perl emit panic warnings such > as the following ever since 5.19.7: > > panic: attempt to copy freed scalar 7fb2264ffde0 to 7fb2264ffd08 at > /Users/mschout/Sync/Apache-TryCatch/blib/lib/Apache/TryCatch.pm line 20. > > I've also seen random segementation faults caused by innocent looking > "return" statements in a large code base. > [...] > > I bisected against perl git, using mod_perl 2.0.9 and apache 2.2.29 as > the mod_perl environment, and installing TryCatch from cpan, and > discovered that the following perl commit is the culprit: > >> commit 437e3a7dac994ebace1195549170c81f474d9c20 >> Author: Matthew Horsfall >> Date: Wed Dec 11 18:28:21 2013 -0500 >> >> Optimise out PUSHMARK/RETURN if return is the last statement in a sub. >> >> This makes: >> >> sub baz { return $cat; } >> >> Behave like: >> >> sub baz { $cat; } >> >> Which is notably faster. >> Thanks for the detective work, Michael. I've logged this on rt.perl.org since it isn't yet clear whether the optimization has a problem, or whether mod_perl (or maybe TryCatch / Devel::Declare) is doing anything naughty: https://rt.perl.org/Ticket/Display.html?id=126070
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
On 9/16/15 9:22 AM, Vincent Veyron wrote: > apt-get install apache2-mpm-prefork libapache2-request-perl > libapache2-mod-perl2 libapache2-mod-apreq2 How did you install TryCatch (the only dependency my module uses)? Regards, Michael Schout
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
On Wed, 16 Sep 2015 08:33:33 -0500 Michael Schout wrote: > Are you using debian packages for the CPAN modules at all? Yes, this is what I use : apt-get install apache2-mpm-prefork libapache2-request-perl libapache2-mod-perl2 libapache2-mod-apreq2 I doubt this makes a difference, but I also have use lib qw(/home/lib); in startup.pl, and put your module in this library. Other than that I did not change anything. -- Bien à vous, Vincent Veyron https://legalcase.libremen.com/ Legal case, contract and insurance claim management software
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
I use the Debian apreq package from apt, to install mod_perl, but the rest of my dependencies I install through cpan On Sep 16, 2015 9:33 AM, "Michael Schout" wrote: > On 9/16/15 7:08 AM, Vincent Veyron wrote: > > > I put your test handler on a machine running Debian 8 (jessie), it works > just fine. > > Fails for me on both CentOS 7 and Darwin (OS X 10.10.5). I see you are > using Apache 2.4 (I was using 2.2). I tested with Apache 2.4.16 and it > also fails for me on that as well > > Server version: Apache/2.4.16 (Unix) > Server built: Sep 16 2015 09:21:11 > > I built perl myself to rule out vendor applied patches to the perl dist. > I wonder if one of the many patches debian applies to perl works around > this issue somehow. Debian also applies a bunch of patches (18 as of > current dpkg src) to the mod_perl package. Maybe one of those works > around the issue. I'll sift through those patches and see if any of > those help. > > Are you using debian packages for the CPAN modules at all? Or did you > install all CPAN packages using a CPAN client? Because I also wonder if > they patched TryCatch, Devel::Declare, or one of its dependencies to > work around this issue. > > Here is the failure on CentOS 7, apache 2.4, perl 5.20.3, mod_perl > 2.0.9, all built from scratch, no vendor packages used: > > [mschout@centos7 modperl-trycatch-bug]$ make test > /home/mschout/perlbug/bin/perl -Iblib/arch -Iblib/lib \ > t/TEST -clean > [warning] setting ulimit to allow core files > ulimit -c unlimited; /home/mschout/perlbug/bin/perl > /home/mschout/modperl-trycatch-bug/t/TEST -clean > APACHE_TEST_APXS= APACHE_TEST_GROUP= APACHE_TEST_HTTPD= > APACHE_TEST_PORT= APACHE_TEST_USER= \ > /home/mschout/perlbug/bin/perl -Iblib/arch -Iblib/lib \ > t/TEST -bugreport -verbose=0 > [warning] setting ulimit to allow core files > ulimit -c unlimited; /home/mschout/perlbug/bin/perl > /home/mschout/modperl-trycatch-bug/t/TEST -bugreport -verbose=0 > /home/mschout/perlbug/bin/httpd -d /home/mschout/modperl-trycatch-bug/t > -f /home/mschout/modperl-trycatch-bug/t/conf/httpd.conf -D APACHE2 -D > APACHE2_4 -D PERL_USEITHREADS > using Apache/2.4.16 (prefork MPM) > > waiting 60 seconds for server to start: .. > waiting 60 seconds for server to start: ok (waited 1 secs) > server localhost:8529 started > t/crash.t .. Dubious, test returned 2 (wstat 512, 0x200) > Failed 2/2 subtests > > t/logs/error_log > [Wed Sep 16 09:27:19.394249 2015] [mpm_prefork:notice] [pid 4129] > AH00163: Apache/2.4.16 (Unix) mod_perl/2.0.9 Perl/v5.20.3 configured -- > resuming normal operations > [Wed Sep 16 09:27:19.394372 2015] [mpm_prefork:info] [pid 4129] AH00164: > Server built: Sep 16 2015 09:21:11 > [Wed Sep 16 09:27:19.394381 2015] [core:notice] [pid 4129] AH00094: > Command line: '/home/mschout/perlbug/bin/httpd -d > /home/mschout/modperl-trycatch-bug/t -f > /home/mschout/modperl-trycatch-bug/t/conf/httpd.conf -D APACHE2 -D > APACHE2_4 -D PERL_USEITHREADS' > [Wed Sep 16 09:27:19.394386 2015] [mpm_prefork:debug] [pid 4129] > prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) > [Wed Sep 16 09:27:20.246297 2015] [authz_core:debug] [pid 4130] > mod_authz_core.c(835): [client 127.0.0.1:46738] AH01628: authorization > result: granted (no directives) > [Wed Sep 16 09:27:20.498088 2015] [authz_core:debug] [pid 4130] > mod_authz_core.c(835): [client 127.0.0.1:46739] AH01628: authorization > result: granted (no directives) > [Wed Sep 16 09:27:20.500796 2015] [perl:error] [pid 4130] [client > 127.0.0.1:46739] panic: attempt to copy freed scalar 34259c0 to 3425870 > at /home/mschout/modperl-trycatch-bug/blib/lib/Apache/TryCatch.pm line > 20.\n > [Wed Sep 16 09:27:20.526382 2015] [core:info] [pid 4129] AH00096: > removed PID file /home/mschout/modperl-trycatch-bug/t/logs/httpd.pid > (pid=4129) > [Wed Sep 16 09:27:20.526398 2015] [mpm_prefork:notice] [pid 4129] > AH00169: caught SIGTERM, shutting down > > > But you are likely not seeing it because one of: > > - vendor patches applied by debian packaging team > - you are using Apache 2.4. I tested with Apache 2.2. > > > Regards, > Michael Schout >
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
On 9/16/15 7:08 AM, Vincent Veyron wrote: > I put your test handler on a machine running Debian 8 (jessie), it works just > fine. Fails for me on both CentOS 7 and Darwin (OS X 10.10.5). I see you are using Apache 2.4 (I was using 2.2). I tested with Apache 2.4.16 and it also fails for me on that as well Server version: Apache/2.4.16 (Unix) Server built: Sep 16 2015 09:21:11 I built perl myself to rule out vendor applied patches to the perl dist. I wonder if one of the many patches debian applies to perl works around this issue somehow. Debian also applies a bunch of patches (18 as of current dpkg src) to the mod_perl package. Maybe one of those works around the issue. I'll sift through those patches and see if any of those help. Are you using debian packages for the CPAN modules at all? Or did you install all CPAN packages using a CPAN client? Because I also wonder if they patched TryCatch, Devel::Declare, or one of its dependencies to work around this issue. Here is the failure on CentOS 7, apache 2.4, perl 5.20.3, mod_perl 2.0.9, all built from scratch, no vendor packages used: [mschout@centos7 modperl-trycatch-bug]$ make test /home/mschout/perlbug/bin/perl -Iblib/arch -Iblib/lib \ t/TEST -clean [warning] setting ulimit to allow core files ulimit -c unlimited; /home/mschout/perlbug/bin/perl /home/mschout/modperl-trycatch-bug/t/TEST -clean APACHE_TEST_APXS= APACHE_TEST_GROUP= APACHE_TEST_HTTPD= APACHE_TEST_PORT= APACHE_TEST_USER= \ /home/mschout/perlbug/bin/perl -Iblib/arch -Iblib/lib \ t/TEST -bugreport -verbose=0 [warning] setting ulimit to allow core files ulimit -c unlimited; /home/mschout/perlbug/bin/perl /home/mschout/modperl-trycatch-bug/t/TEST -bugreport -verbose=0 /home/mschout/perlbug/bin/httpd -d /home/mschout/modperl-trycatch-bug/t -f /home/mschout/modperl-trycatch-bug/t/conf/httpd.conf -D APACHE2 -D APACHE2_4 -D PERL_USEITHREADS using Apache/2.4.16 (prefork MPM) waiting 60 seconds for server to start: .. waiting 60 seconds for server to start: ok (waited 1 secs) server localhost:8529 started t/crash.t .. Dubious, test returned 2 (wstat 512, 0x200) Failed 2/2 subtests t/logs/error_log [Wed Sep 16 09:27:19.394249 2015] [mpm_prefork:notice] [pid 4129] AH00163: Apache/2.4.16 (Unix) mod_perl/2.0.9 Perl/v5.20.3 configured -- resuming normal operations [Wed Sep 16 09:27:19.394372 2015] [mpm_prefork:info] [pid 4129] AH00164: Server built: Sep 16 2015 09:21:11 [Wed Sep 16 09:27:19.394381 2015] [core:notice] [pid 4129] AH00094: Command line: '/home/mschout/perlbug/bin/httpd -d /home/mschout/modperl-trycatch-bug/t -f /home/mschout/modperl-trycatch-bug/t/conf/httpd.conf -D APACHE2 -D APACHE2_4 -D PERL_USEITHREADS' [Wed Sep 16 09:27:19.394386 2015] [mpm_prefork:debug] [pid 4129] prefork.c(995): AH00165: Accept mutex: sysvsem (default: sysvsem) [Wed Sep 16 09:27:20.246297 2015] [authz_core:debug] [pid 4130] mod_authz_core.c(835): [client 127.0.0.1:46738] AH01628: authorization result: granted (no directives) [Wed Sep 16 09:27:20.498088 2015] [authz_core:debug] [pid 4130] mod_authz_core.c(835): [client 127.0.0.1:46739] AH01628: authorization result: granted (no directives) [Wed Sep 16 09:27:20.500796 2015] [perl:error] [pid 4130] [client 127.0.0.1:46739] panic: attempt to copy freed scalar 34259c0 to 3425870 at /home/mschout/modperl-trycatch-bug/blib/lib/Apache/TryCatch.pm line 20.\n [Wed Sep 16 09:27:20.526382 2015] [core:info] [pid 4129] AH00096: removed PID file /home/mschout/modperl-trycatch-bug/t/logs/httpd.pid (pid=4129) [Wed Sep 16 09:27:20.526398 2015] [mpm_prefork:notice] [pid 4129] AH00169: caught SIGTERM, shutting down But you are likely not seeing it because one of: - vendor patches applied by debian packaging team - you are using Apache 2.4. I tested with Apache 2.2. Regards, Michael Schout
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
On Tue, 15 Sep 2015 17:18:06 -0500 Michael Schout wrote: > On 9/15/15 5:15 PM, Michael Schout wrote: > > > Reverting the change to op.c completely fixes the problem. I cannot > > reproduce the panic after this change, and I do not get any segmentation > > faults either in a large codebase that I am also testing against. > > I should have mentioned I reverted the change in 5.19.7, and also 5.20.3 > and in both cases the problem is gone. > Hi Michael, I put your test handler on a machine running Debian 8 (jessie), it works just fine. perl -v : This is perl 5, version 20, subversion 2 (v5.20.2) built for x86_64-linux-gnu-thread-multi dpkg -l *mod-perl* : ii libapache2-mod-perl2 2.0.9~1624218-2 amd64 Integration of perl with the Apache2 web server apache2 -v : Server version: Apache/2.4.10 (Debian) Server built: Aug 1 2015 20:53:57 -- Bien à vous, Vincent Veyron https://legalcase.libremen.com/ Legal case, contract and insurance claim management software
Re: panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
On 9/15/15 5:15 PM, Michael Schout wrote: > Reverting the change to op.c completely fixes the problem. I cannot > reproduce the panic after this change, and I do not get any segmentation > faults either in a large codebase that I am also testing against. I should have mentioned I reverted the change in 5.19.7, and also 5.20.3 and in both cases the problem is gone. Regards, Michael Schout
panic: attempt to copy freed scalar caused by calling cgi->param with TryCatch since perl 5.19.7
As reported in other thread, I've seen mod_perl emit panic warnings such as the following ever since 5.19.7: panic: attempt to copy freed scalar 7fb2264ffde0 to 7fb2264ffd08 at /Users/mschout/Sync/Apache-TryCatch/blib/lib/Apache/TryCatch.pm line 20. I've also seen random segementation faults caused by innocent looking "return" statements in a large code base. I've created a test mod perl handler that demonstrates this behaviour: -- package Apache::TryCatch; use Apache2::Const -compile => qw(OK); use TryCatch; use CGI; sub handler :method { my ($self, $r) = @_; $r->content_type('text/plain'); print $self->crash($r); return Apache2::Const::OK; } sub crash { my $cgi = CGI->new; try { $cgi->param(x => 'x') } return "Hello World!"; } 1; -- configured in httpd.conf as: PerlModule Apache::TryCatch PerlSwitches -I@ServerRoot/../lib SetHandler perl-script PerlResponseHandler Apache::TryCatch -- Simply accessing /crashme causes a 500 internal server error and the panic warning in the httpd error log. I bisected against perl git, using mod_perl 2.0.9 and apache 2.2.29 as the mod_perl environment, and installing TryCatch from cpan, and discovered that the following perl commit is the culprit: > commit 437e3a7dac994ebace1195549170c81f474d9c20 > Author: Matthew Horsfall > Date: Wed Dec 11 18:28:21 2013 -0500 > > Optimise out PUSHMARK/RETURN if return is the last statement in a sub. > > This makes: > > sub baz { return $cat; } > > Behave like: > > sub baz { $cat; } > > Which is notably faster. > > Unpatched: > > ./perl -Ilib/ ~/stuff/bench.pl > Benchmark: timing 4000 iterations of normal, ret... > normal: 3 wallclock secs ( 1.60 usr + 0.01 sys = 1.61 CPU) @ > 24844720.50/s (n=4000) >ret: 3 wallclock secs ( 2.08 usr + 0.00 sys = 2.08 CPU) @ > 19230769.23/s (n=4000) > > Patched: > > ./perl -Ilib ~/stuff/bench.pl > Benchmark: timing 4000 iterations of aret, normal... > normal: 2 wallclock secs ( 1.72 usr + 0.00 sys = 1.72 CPU) @ > 23255813.95/s (n=4000) >ret: 2 wallclock secs ( 1.72 usr + 0.00 sys = 1.72 CPU) @ > 23255813.95/s (n=4000) > > The difference in OP trees can be seen here: > > Unpatched: > > $ perl -MO=Concise,baz -e 'sub baz { return $cat }' > main::baz: > 5 <1> leavesub[1 ref] K/REFC,1 ->(end) > - <@> lineseq KP ->5 > 1<;> nextstate(main 1 -e:1) v ->2 > 4<@> return K ->5 > 2 <0> pushmark s ->3 > - <1> ex-rv2sv sK/1 ->4 > 3 <#> gvsv[*cat] s ->4 > -e syntax OK > > Patched: > > $ ./perl -Ilib -MO=Concise,baz -e 'sub baz { return $cat }' > main::baz: > 3 <1> leavesub[1 ref] K/REFC,1 ->(end) > - <@> lineseq KP ->3 > 1<;> nextstate(main 1 -e:1) v ->2 > -<@> return K ->- > - <0> pushmark s ->2 > - <1> ex-rv2sv sK/1 ->- > 2 <$> gvsv(*cat) s ->3 > -e syntax OK > > (Includes some modifications from Steffen) > > ext/B/t/optree_samples.t | 18 +- > op.c | 31 +++ > op.h | 6 ++ > 3 files changed, 54 insertions(+), 1 deletion(-) Reverting the change to op.c completely fixes the problem. I cannot reproduce the panic after this change, and I do not get any segmentation faults either in a large codebase that I am also testing against. This problem seems to be mod_perl specific as the try { $cgi->param() } does not cause a panic when run outside of mod_perl. In addition, not all try { } blocks cause the panic. It seems that calling CGI.pm's param() method to set a value *does* cause it however. If anyone has any ideas on what might be causing behaviour in mod_perl, I'm all ears. I may try to dig into this, but I'm not familiar with mod_perl internals so that may take quite some time. Admittedly, TryCatch depends on Devel::Declare, which as I understand it makes changes to the optree at runtime. I'm guessing there is some interaction with the optree in mod_perl that TryCatch along with perl's optimization that removes the "return" ops that is causing this. I have put the test case as a module up on github so that anyone can clone it to reproduce the bug. The repository is here: https://github.com/mschout/modperl-trycatch-bug.git The readme explains how to run it, but essentially its install a perl >= 5.19.7, install mod_perl, install TryCatch, and then the usual "perl Makefile.PL && make test" Regards, Michael Schout