4000 is the default on the Centos5 servers I have installed. That's why I was surprised by the number. This server is Debian, maybe that default is that different? In any event, I agree with you - someone probably put it there for a reason so I would leave it alone until I had a good reason to do otherwise. Not sure I am willing to start trying different settings. This particular server hosts a lot of customers and we will be moving them off to virtualized OpenVZ VE's over the next few months. Since even this incident doesn't really disrupt service that dramatically, I am inclined to just hold tight and keep gathering more information.
----- Original Message ----- From: "Nick Floersch" <[EMAIL PROTECTED]> To: [email protected] Sent: Thursday, May 8, 2008 6:26:51 PM GMT -05:00 US/Canada Eastern Subject: Re: Apache Problem Sorry for the late reply - I didn't see this when it came in... Honestly, IANAAG - I am not an Apache guru. I remember from my UNIX TCP/IP programming course that we discussed the entire worker/thread/pre-spawn model that Apache uses when we talked about writing servers. That said, I haven't examined statistics to know what great numbers are to plug into those settings. However, I would compare what you have to the defaults ... is 100000 a default? I don't know, but if it is, then it may not be a bad choice. If some previous SignalZ admin put that in there, unless it was Matt C. (who is the only SignalZ admin I knew well enough to know to trust his every action), I would research the number and why it was set to that. I personally would play with the numbers, too. Try setting the MaxRequestsPerChild to 100. See what happens. There must be lots of articles on this subject somewhere. What about the MaxKeepAlives and KeepAliveTimeout directives? Or just the Timeout directive? Perhaps another problem is that a lot of these threads are getting spawned and then just taking too long to go away - if someone mucked up those settings then you might see the effect you are seeing. Lastly, I tried searching Google for '..reading..' to see what is out there. The only thing I found was a single post saying "if you are seeing a lot of log entries like this, then you might be experiencing a DOS" and then it discussed firewall setup. Unfortunately, it seems that most if not all search engines refuse to parse a period, or pair of periods for that matter - they just get dropped. So searching for '..reading..' is the same as searching for 'reading', which does not help narrow down the hits. http://www.cpanelconfig.com/cpanel-security-related-articles/apache-read ing-dos-attack/ Which I could be more help. -Nick ________________________________ From: Vermont Area Group of Unix Enthusiasts [mailto:[EMAIL PROTECTED] On Behalf Of Sue Fritz Sent: Thursday, May 08, 2008 3:35 PM To: [email protected] Subject: Re: Apache Problem Nick, this might be heading me in the right direction. How do you like the looks of these settings from my apache config? StartServers 10 MinSpareServers 5 MaxSpareServers 30 MaxClients 255 MaxRequestsPerChild 100000 I am guessing that the MaxRequestsPerChild might be a little high, ya think? "The MaxRequestsPerChild directive defines the maximum number of page deliveries that each server instance will carry out before closing down and respawning. The whole point of a periodic respawn is to prevent accumulation of eventual memory leaks. " Sue Fritz system admin Signal Advertising ----- Original Message ----- From: "Nick Floersch" <[EMAIL PROTECTED]> To: [email protected] Sent: Thursday, May 8, 2008 2:35:17 PM GMT -05:00 US/Canada Eastern Subject: Re: Apache Problem Phil, this is the Apache config stuff I was thinking of. It *might* be helpful only in controlling what your server does with itself when lots of connections start rolling in. If The 255 limit is eating up memory needed by other servers or processes, you could trim back that limit to allow other stuff to use the resources, for example. Here is an example from my config file: --- snip --- ## ## Server-Pool Size Regulation (MPM specific) ## # prefork MPM # StartServers ......... number of server processes to start # MinSpareServers ...... minimum number of server processes which are kept spare # MaxSpareServers ...... maximum number of server processes which are kept spare # MaxClients ........... maximum number of server processes allowed to start # MaxRequestsPerChild .. maximum number of requests a server process serves <IfModule prefork.c> StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 20 MaxRequestsPerChild 0 </IfModule> # pthread MPM # StartServers ......... initial number of server processes to start # MaxClients ........... maximum number of server processes allowed to start # MinSpareThreads ...... minimum number of worker threads which are kept spare # MaxSpareThreads ...... maximum number of worker threads which are kept spare # ThreadsPerChild ...... constant number of worker threads in each server process # MaxRequestsPerChild .. maximum number of requests a server process serves <IfModule worker.c> StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 </IfModule> --- snip --- So, those are examples. The docs are at this URL: http://httpd.apache.org/docs/2.2/mod/mpm_common.html#serverlimit But you may also want to look for your timeout directives to decrease how long those threads sit waiting. Also, the KeepAlive settings may be fouling up somehow, and worth changing a bit. --- snip --- Timeout 300 KeepAlive On MaxKeepAliveRequests 100 KeepAliveTimeout 15 --- snip --- The URL for that stuff is in the Core directive docs. http://httpd.apache.org/docs/2.2/mod/core.html#timeout http://httpd.apache.org/docs/2.2/mod/core.html#keepalive http://httpd.apache.org/docs/2.2/mod/core.html#maxkeepaliverequests http://httpd.apache.org/docs/2.2/mod/core.html#keepalivetimeout Hopefully something here could help tune Apache to handle these requests better. I can't help much more than you know on blocking the requests, however. I've never seen log entries like yours where there is no IP address recorded for the connection. That seems pretty odd to me. -Nick -----Original Message----- From: Vermont Area Group of Unix Enthusiasts [mailto:[EMAIL PROTECTED] On Behalf Of Nick Floersch Sent: Thursday, May 08, 2008 1:37 PM To: [email protected] Subject: Re: Apache Problem Have you ruled out a deliberate DOS attack? How quickly does it shoot to 255 processes? There is an option to increase the number of child threads the master can split off... but I suspect that increasing the limit would just provide more room for it to fill up. There might be some way to limit the number of connections per client. Still, the Apache worker thread controls might be helpful to read about here. Are all of the connections from the same client IP? Or are you getting actions from lots of IPs where the child threads don't quit after finishing the transactions? There is a program called Apache Top, sort of like regular UNIX Top... but maybe it would help monitor this stuff? Or is that you are using? Are these threads initiating processes which interact with a database at all? Could the database connections not be closing cleanly, leaving Apache thinking it is still reading/writing/connected? What server side scripting are you using? PHP, Perl, Python, Ruby? What do the logs say? -Nick -----Original Message----- From: Vermont Area Group of Unix Enthusiasts [mailto:[EMAIL PROTECTED] On Behalf Of Phil Marshall Sent: Thursday, May 08, 2008 1:23 PM To: [email protected] Subject: Apache Problem Here is a sample from the apache status output. We are getting flooded with these. Apache will get filled up to 255 connections with these things. This has happened before, but usually it goes away quickly. Today it went on for about 45 minutes. Its fine now. Does anyone have any idea what this is or how to block it? 51-9 28349 0/1/1 R 0.00 28 3 0.0 0.02 0.02 ? ? ..reading.. 52-9 28350 0/1/1 R 0.00 28 0 0.0 0.000 0.000 ? ? ..reading.. 53-9 28351 0/2/2 R 0.00 28 0 0.0 0.000 0.000 ? ? ..reading.. 54-9 28352 0/1/1 R 0.00 28 0 0.0 0.000 0.000 ? ? ..reading.. 55-9 28354 0/1/1 R 0.00 28 78 0.0 0.00 0.00 ? ? ..reading.. 56-9 28355 0/1/1 R 0.00 28 0 0.0 0.000 0.000 ? ? ..reading.. --------------------- Phil Marshall Signal Advertising 535 Stone Cutters Way Montpelier, VT 05602 802-229-4149 [EMAIL PROTECTED]
