Sorry I dropped off - yesterday Phil and I went to do some work on our rack in Waltham.
I am not 100% sure how much you got from Phil - we just recently started testing an anti-spoofing filter on our Postini account and now neither Phil or I are receiving our own posts to this list. I knew this kind of thing would happen, just didn't realize how much stuff out there actually validly spoofs. So, for the sake of filling in the missing pieces on the apache issue. We have been having this issue very intermittently for about 4-5 months. In the past, it would start and go away so fast I couldn't gather any info. We have a huge volume of logs on this server and I wasn't quite sure where to start looking without having the opportunity to get an IP to narrow the search. This time, it lasted long enough for me to do a netstat and find one IP that had over 500 connections open. In the logs, that IP generated an equally large number of timeout errors that look like this.... "-" 408 - "-" "-" The only web activity reported for that IP is to this site http://www.thehanovertheatre.org/. So at this point I am left with a little more information, but maybe not enough to really know anything for sure. I might suspect something wrong with the code on this page - maybe some glitch in the page causes the java script to blast off a lot of connections??? I am not sure how much I pursue that angle until I see it happen again in the same way. I did add code to dump netstats everytime this event occurs - not sure why I didn't think of that before. On the note about the apache settings. I am new to linux, but as a general philosophy I don't like the "unbounded" approach. In my past experiences, if one of my webservers croaked, it croaked because someone put sloppy or untested code on it. We're all human, right? Anything I can do to mitigate the affects of mistakes is something I want to do. With that said, I know little about tuning linux/apache, and since most of Signals stuff works well, I haven't spent a lot of time learning about this stuff. sue ----- Original Message ----- From: "Rene Churchill" <[EMAIL PROTECTED]> To: [email protected] Sent: Thursday, May 8, 2008 7:33:36 PM GMT -05:00 US/Canada Eastern Subject: Re: Apache Problem Sue, Phil, We need to see some snippets of the access_log and error_log logfiles to have a shot at figuring this out. As for the config options you've got, those are pretty good for the vast majority of servers out there. MaxRequestsPerChild is there to prevent memory leaks. Since Apache often load many, many modules written by folks other than the Apache team, they don't have the highest confidence in their garbage collection. So it makes sense to restart the process every once in a while to free up any leaked memory. Don't bother changing it unless you see the Apache processes chewing up more and more memory over time. Since your current problem is too many Apache processes, tweaking MaxRequestsPerChild won't help any. Personally, I expect to see the logfiles showing a lot of unusual HTTP requests, perhaps malformed. Googling the browser string may be of some use. If someone is just opening a connection to port 80 and letting it timeout, then playing with the TimeOut directive may help. Also consider disabling KeepAlive. http://httpd.apache.org/docs/1.3/mod/core.html#timeout http://httpd.apache.org/docs/1.3/mod/core.html#keepalive Hope that helps, Rene Sue Fritz wrote: > Nick, this might be heading me in the right direction. How do you like > the looks of these settings from my apache config? > > StartServers 10 > MinSpareServers 5 > MaxSpareServers 30 > MaxClients 255 > xMaxRequestsPerChild 100000 > > I am guessing that the MaxRequestsPerChild might be a little high, ya think? > > "The MaxRequestsPerChild directive defines the maximum number of page > deliveries that each server instance will carry out before closing down > and respawning. The whole point of a periodic respawn is to prevent > accumulation of eventual memory leaks. " > > > Sue Fritz > system admin > Signal Advertising > > ----- Original Message ----- > From: "Nick Floersch" <[EMAIL PROTECTED]> > To: [email protected] > Sent: Thursday, May 8, 2008 2:35:17 PM GMT -05:00 US/Canada Eastern > Subject: Re: Apache Problem > > Phil, this is the Apache config stuff I was thinking of. It *might* be > helpful only in controlling what your server does with itself when lots > of connections start rolling in. If The 255 limit is eating up memory > needed by other servers or processes, you could trim back that limit to > allow other stuff to use the resources, for example. > > Here is an example from my config file: > --- snip --- > ## > ## Server-Pool Size Regulation (MPM specific) > ## > > # prefork MPM > # StartServers ......... number of server processes to start > # MinSpareServers ...... minimum number of server processes which are > kept spare > # MaxSpareServers ...... maximum number of server processes which are > kept spare > # MaxClients ........... maximum number of server processes allowed to > start > # MaxRequestsPerChild .. maximum number of requests a server process > serves > <IfModule prefork.c> > StartServers 5 > MinSpareServers 5 > MaxSpareServers 10 > MaxClients 20 > MaxRequestsPerChild 0 > </IfModule> > > # pthread MPM > # StartServers ......... initial number of server processes to start > # MaxClients ........... maximum number of server processes allowed to > start > # MinSpareThreads ...... minimum number of worker threads which are > kept spare > # MaxSpareThreads ...... maximum number of worker threads which are > kept spare > # ThreadsPerChild ...... constant number of worker threads in each > server process > # MaxRequestsPerChild .. maximum number of requests a server process > serves > <IfModule worker.c> > StartServers 2 > MaxClients 150 > MinSpareThreads 25 > MaxSpareThreads 75 > ThreadsPerChild 25 > MaxRequestsPerChild 0 > </IfModule> > > --- snip --- > So, those are examples. The docs are at this URL: > http://httpd.apache.org/docs/2.2/mod/mpm_common.html#serverlimit > > But you may also want to look for your timeout directives to decrease > how long those threads sit waiting. Also, the KeepAlive settings may be > fouling up somehow, and worth changing a bit. > --- snip --- > Timeout 300 > > KeepAlive On > > MaxKeepAliveRequests 100 > > KeepAliveTimeout 15 > --- snip --- > > The URL for that stuff is in the Core directive docs. > http://httpd.apache.org/docs/2.2/mod/core.html#timeout > > http://httpd.apache.org/docs/2.2/mod/core.html#keepalive > > http://httpd.apache.org/docs/2.2/mod/core.html#maxkeepaliverequests > > http://httpd.apache.org/docs/2.2/mod/core.html#keepalivetimeout > > > Hopefully something here could help tune Apache to handle these requests > better. I can't help much more than you know on blocking the requests, > however. I've never seen log entries like yours where there is no IP > address recorded for the connection. That seems pretty odd to me. > > -Nick > > -----Original Message----- > From: Vermont Area Group of Unix Enthusiasts [mailto:[EMAIL PROTECTED] > On Behalf Of Nick Floersch > Sent: Thursday, May 08, 2008 1:37 PM > To: [email protected] > Subject: Re: Apache Problem > > Have you ruled out a deliberate DOS attack? > > How quickly does it shoot to 255 processes? There is an option to > increase the number of child threads the master can split off... but I > suspect that increasing the limit would just provide more room for it to > fill up. There might be some way to limit the number of connections per > client. Still, the Apache worker thread controls might be helpful to > read about here. > > Are all of the connections from the same client IP? Or are you getting > actions from lots of IPs where the child threads don't quit after > finishing the transactions? > > There is a program called Apache Top, sort of like regular UNIX Top... > but maybe it would help monitor this stuff? Or is that you are using? > > Are these threads initiating processes which interact with a database at > all? Could the database connections not be closing cleanly, leaving > Apache thinking it is still reading/writing/connected? > > What server side scripting are you using? PHP, Perl, Python, Ruby? > > What do the logs say? > > -Nick > > -----Original Message----- > From: Vermont Area Group of Unix Enthusiasts [mailto:[EMAIL PROTECTED] > On Behalf Of Phil Marshall > Sent: Thursday, May 08, 2008 1:23 PM > To: [email protected] > Subject: Apache Problem > > Here is a sample from the apache status output. We are getting > flooded with these. Apache will get filled up to 255 connections with > these things. This has happened before, but usually it goes away > quickly. Today it went on for about 45 minutes. Its fine now. Does > anyone have any idea what this is or how to block it? > > 51-9 28349 0/1/1 R 0.00 28 3 0.0 0.02 > 0.02 ? ? ..reading.. > 52-9 28350 0/1/1 R 0.00 28 0 0.0 0.000 > 0.000 ? ? ..reading.. > 53-9 28351 0/2/2 R 0.00 28 0 0.0 0.000 > 0.000 ? ? ..reading.. > 54-9 28352 0/1/1 R 0.00 28 0 0.0 0.000 > 0.000 ? ? ..reading.. > 55-9 28354 0/1/1 R 0.00 28 78 0.0 0.00 > 0.00 ? ? ..reading.. > 56-9 28355 0/1/1 R 0.00 28 0 0.0 0.000 > 0.000 ? ? ..reading.. > > --------------------- > Phil Marshall > Signal Advertising > 535 Stone Cutters Way > Montpelier, VT 05602 > 802-229-4149 > [EMAIL PROTECTED]
