Synopsis: My servers, installed per instructions at http://www.peregrinehw.com/downloads/apached/devel, were originally compiled in July of Last year. On February 6th of this year, without warning or change, we began segfault 11'ing for the first time in years on multiple servers with enough differences to rule out code bases or hardware. On Mar 26th, we turned off mod_perl by SIMPLY commenting the Apache::Registry line in httpd.conf and NOT recompiling. Our segfaults stopped.
After more research I thought it was a bug, exploit or attack caused by a malicious program or user sending a bizarre string. To test this, I enabled a VERY strict .htaccess on our development site denying access to all but our beta tests. The segfaults stopped even with mod_perl enabled again. Now, more than 2 months later, we are still working on why and how to fix the problem. Through the recommendation of many people at the Apache project and the PHP exploit, we have upgraded to Apache 1.3.23. We are now trying Apache 1.3.24 as of 3PM today. We have tried in vain to get a core file (Compiled with the #WITH DEBUG lines in the instructions above and chmod'd 777 core files in /usr/local/apache. However, the child processes don't actually core dump so their is nothing to trace. If anyone can help me get a core file, I think this would help immensely. So, the best I can do is read and try changes one and a time and I am ready to make this crackpot theory ;-) Code Red II (or a variant thereof) starts at octet 63. My servers are at 66 and it has taken till February to get there. Hence, the delay between the compilation in July and the segfaults in Feb and it explains why my servers are bombing and not hundreds of others on different IP ranges. What's odd is I thought a follow-up request for default.ida was part of Code Red but I'm not seeing those errors, just the malformed host header Anyway, I believe we have now correlated a malformed host entry with our segfaults finally. It takes a while but this is what seems to blow it up. It ONLY happens when Apache::Registry is enabled in the httpd.conf (still compiled in and still loading startup.pl, just no scripts are activating it). [Mon Apr 8 14:04:03 2002] [error] [client 195.210.129.26] Client sent malformed Host header [Mon Apr 8 14:12:51 2002] [notice] child pid 11889 exit signal Segmentation fault (11) [Mon Apr 8 15:04:49 2002] [error] [client 218.76.7.137] Client sent malformed Host header [Mon Apr 8 15:42:52 2002] [notice] child pid 13768 exit signal Segmentation fault (11) As you can see, it takes a while to crash the process but blocking access to the server via .htaccess STOPS the segmentation faults as I mentioned before. I would appreciate any comments of similar experiences or help in regards to making a core file. Regards, KAM
