Re: Help Requested: Segfault 11 7 MONTHS after compilation on multiple servers all compiled the same running different code and different Redhat Released all on the same day [BUG]
Stas, I'm much closer thanks to your document*. I now at least I feel like I'm moving forward for the first time in months. OK, so I'm pretty sure short of recompiling Perl, that I have all the correct debug things turned on though the user permission problem still plague me a bit. I just chgrp'd the whole dir to httpd and did a chmod g+w -R so I think that should fix any permission problems for a core file to be created. Here's some notes abour your document and the problems I had as I'm pretty sure it's a work in progress. 1st, I had a little trouble following the Bad Segv stuff because I had to install Parser and then Inline. That was no big deal but then your code was written for a much older version of Inline and no longer works, yada, yada, yada. The version of Bad::Segv that I developed VERY quickly to get around this is up at http://www.peregrinehw.com/downloads/apache/mod_perl 2nd, The segv.cgi at the same location as the Bad:Segv above, segfaults on the command line and through normal CGI but not with mod_perl. Here's the error I get trying to run this script with mod_perl (1.26): [Tue Apr 9 09:53:16 2002] [error] [Tue Apr 9 09:53:16 2002] PerlHandler subroutine `Apache::Registry::handler': Undefined subroutine Bad::Segv::segv called at /htdocs/peregrinehw.com/html/segv.cgim line 12. [Tue Apr 9 09:53:16 2002] END block subroutine `Inline::END': One or moreDATA sections were not processed by Inline. [Tue Apr 9 09:53:16 2002] END block subroutine `Inline::END': Thoughts? Something special with mod_perl? I didn't add the Bad::Segv to the startup.pl or anything at all. Regards, Kevin A. McGrail * http://perl.apache.org/preview/modperl-docs/dst_html/docs/2.0/devel/debug_c/ debug_c.html#Getting_the_core_File_Dumped Bad-Segv-0.20.tar.gz Description: GNU Zip compressed data
Re: Help Requested: Segfault 11 7 MONTHS after compilation on multiple servers all compiled the same running different code and different Redhat Released all on the same day [BUG]
Kevin A. McGrail wrote: 2nd, The segv.cgi at the same location as the Bad:Segv above, segfaults on the command line and through normal CGI but not with mod_perl. Here's the error I get trying to run this script with mod_perl (1.26): [Tue Apr 9 09:53:16 2002] [error] [Tue Apr 9 09:53:16 2002] PerlHandler subroutine `Apache::Registry::handler': Undefined subroutine Bad::Segv::segv called at /htdocs/peregrinehw.com/html/segv.cgim line 12. [Tue Apr 9 09:53:16 2002] END block subroutine `Inline::END': One or moreDATA sections were not processed by Inline. [Tue Apr 9 09:53:16 2002] END block subroutine `Inline::END': Thoughts? Something special with mod_perl? I didn't add the Bad::Segv to the startup.pl or anything at all. I haven't been following this thread, but it appears to me from the error message that the DATA section might be causing the problem in conjunction with Apache::Registry, which explicitly states: Your scripts cannot contain the __END__ or __DATA__ token to terminate compilation. in the CAVEATS section. Ignore this if I'm offbase. (darren) -- The knowledge that makes us cherish innocence makes innocence unattainable. -- Irving Howe
Help Requested: Segfault 11 7 MONTHS after compilation on multiple servers all compiled the same running different code and different Redhat Released all on the same day [BUG]
Synopsis: My servers, installed per instructions at http://www.peregrinehw.com/downloads/apached/devel, were originally compiled in July of Last year. On February 6th of this year, without warning or change, we began segfault 11'ing for the first time in years on multiple servers with enough differences to rule out code bases or hardware. On Mar 26th, we turned off mod_perl by SIMPLY commenting the Apache::Registry line in httpd.conf and NOT recompiling. Our segfaults stopped. After more research I thought it was a bug, exploit or attack caused by a malicious program or user sending a bizarre string. To test this, I enabled a VERY strict .htaccess on our development site denying access to all but our beta tests. The segfaults stopped even with mod_perl enabled again. Now, more than 2 months later, we are still working on why and how to fix the problem. Through the recommendation of many people at the Apache project and the PHP exploit, we have upgraded to Apache 1.3.23. We are now trying Apache 1.3.24 as of 3PM today. We have tried in vain to get a core file (Compiled with the #WITH DEBUG lines in the instructions above and chmod'd 777 core files in /usr/local/apache. However, the child processes don't actually core dump so their is nothing to trace. If anyone can help me get a core file, I think this would help immensely. So, the best I can do is read and try changes one and a time and I am ready to make this crackpot theory ;-) Code Red II (or a variant thereof) starts at octet 63. My servers are at 66 and it has taken till February to get there. Hence, the delay between the compilation in July and the segfaults in Feb and it explains why my servers are bombing and not hundreds of others on different IP ranges. What's odd is I thought a follow-up request for default.ida was part of Code Red but I'm not seeing those errors, just the malformed host header Anyway, I believe we have now correlated a malformed host entry with our segfaults finally. It takes a while but this is what seems to blow it up. It ONLY happens when Apache::Registry is enabled in the httpd.conf (still compiled in and still loading startup.pl, just no scripts are activating it). [Mon Apr 8 14:04:03 2002] [error] [client 195.210.129.26] Client sent malformed Host header [Mon Apr 8 14:12:51 2002] [notice] child pid 11889 exit signal Segmentation fault (11) [Mon Apr 8 15:04:49 2002] [error] [client 218.76.7.137] Client sent malformed Host header [Mon Apr 8 15:42:52 2002] [notice] child pid 13768 exit signal Segmentation fault (11) As you can see, it takes a while to crash the process but blocking access to the server via .htaccess STOPS the segmentation faults as I mentioned before. I would appreciate any comments of similar experiences or help in regards to making a core file. Regards, KAM
