Hi Sue,
Ok, 408 is a timeout error, which is indicative of a DOS attack. The
attacker opens a port on your server and just lets the connection
sit there until it times out (default 300 seconds). This ties up a
connection for those 300 seconds without requiring the webserver to do
anything. Max out the connections and nobody can connect to the
webserver.
So, first thing is to look up that IP address. If it's from someplace
relatively untraceable like Romania, then just block it with your
firewall or iptables. If the IP is from someplace you think you'd
have a chance to complain about, go ahead and report it.
You mentioned http://www.thehanovertheatre.org/. Is the IP address
for http://www.thehanovertheatre.org/ where the request is coming
from or where it is going to? I doubt it is a problem with the
code on the page, otherwise you'd be seeing this problem constantly.
Next, edit your httpd.conf file and reduce the Timeout value to
something much smaller, like 5 or 10 seconds.
http://httpd.apache.org/docs/1.3/mod/core.html#timeout
An automated attack can still swamp your webserver with a low
timeout value, but it will show up in the logfiles much sooner
and clean up much faster when you block the IP address. If the
problem does continue, let us know and someone can write up a
script to monitor the apache logfile for 408 errors and block
the naughty IP address w/ iptables.
Hope that helps,
Rene
Sue Fritz wrote:
Sorry I dropped off - yesterday Phil and I went to do some work on our
rack in Waltham.
I am not 100% sure how much you got from Phil - we just recently started
testing an anti-spoofing filter on our Postini account and now neither
Phil or I are receiving our own posts to this list. I knew this kind of
thing would happen, just didn't realize how much stuff out there
actually validly spoofs.
So, for the sake of filling in the missing pieces on the apache issue.
We have been having this issue very intermittently for about 4-5 months.
In the past, it would start and go away so fast I couldn't gather any
info. We have a huge volume of logs on this server and I wasn't quite
sure where to start looking without having the opportunity to get an IP
to narrow the search. This time, it lasted long enough for me to do a
netstat and find one IP that had over 500 connections open. In the logs,
that IP generated an equally large number of timeout errors that look
like this....
"-" 408 - "-" "-"
The only web activity reported for that IP is to this site
http://www.thehanovertheatre.org/.
So at this point I am left with a little more information, but maybe not
enough to really know anything for sure. I might suspect something wrong
with the code on this page - maybe some glitch in the page causes the
java script to blast off a lot of connections??? I am not sure how much
I pursue that angle until I see it happen again in the same way. I did
add code to dump netstats everytime this event occurs - not sure why I
didn't think of that before.
On the note about the apache settings. I am new to linux, but as a
general philosophy I don't like the "unbounded" approach. In my past
experiences, if one of my webservers croaked, it croaked because someone
put sloppy or untested code on it. We're all human, right? Anything I
can do to mitigate the affects of mistakes is something I want to do.
With that said, I know little about tuning linux/apache, and since most
of Signals stuff works well, I haven't spent a lot of time learning
about this stuff.
sue
----- Original Message -----
From: "Rene Churchill" <[EMAIL PROTECTED]>
To: [email protected]
Sent: Thursday, May 8, 2008 7:33:36 PM GMT -05:00 US/Canada Eastern
Subject: Re: Apache Problem
Sue, Phil,
We need to see some snippets of the access_log and error_log
logfiles to have a shot at figuring this out.
As for the config options you've got, those are pretty good for the
vast majority of servers out there.
MaxRequestsPerChild is there to prevent memory leaks. Since Apache
often load many, many modules written by folks other than the Apache
team, they don't have the highest confidence in their garbage
collection. So it makes sense to restart the process every once in
a while to free up any leaked memory. Don't bother changing it unless
you see the Apache processes chewing up more and more memory over time.
Since your current problem is too many Apache processes, tweaking
MaxRequestsPerChild won't help any.
Personally, I expect to see the logfiles showing a lot of unusual
HTTP requests, perhaps malformed. Googling the browser string may
be of some use.
If someone is just opening a connection to port 80 and letting it
timeout, then playing with the TimeOut directive may help. Also
consider disabling KeepAlive.
http://httpd.apache.org/docs/1.3/mod/core.html#timeout
http://httpd.apache.org/docs/1.3/mod/core.html#keepalive
Hope that helps,
Rene
Sue Fritz wrote:
> Nick, this might be heading me in the right direction. How do you like
> the looks of these settings from my apache config?
>
> StartServers 10
> MinSpareServers 5
> MaxSpareServers 30
> MaxClients 255
> xMaxRequestsPerChild 100000
>
> I am guessing that the MaxRequestsPerChild might be a little high, ya
think?
>
> "The MaxRequestsPerChild directive defines the maximum number of page
> deliveries that each server instance will carry out before closing down
> and respawning. The whole point of a periodic respawn is to prevent
> accumulation of eventual memory leaks. "
>
>
> Sue Fritz
> system admin
> Signal Advertising
>
> ----- Original Message -----
> From: "Nick Floersch" <[EMAIL PROTECTED]>
> To: [email protected]
> Sent: Thursday, May 8, 2008 2:35:17 PM GMT -05:00 US/Canada Eastern
> Subject: Re: Apache Problem
>
> Phil, this is the Apache config stuff I was thinking of. It *might* be
> helpful only in controlling what your server does with itself when lots
> of connections start rolling in. If The 255 limit is eating up memory
> needed by other servers or processes, you could trim back that limit to
> allow other stuff to use the resources, for example.
>
> Here is an example from my config file:
> --- snip ---
> ##
> ## Server-Pool Size Regulation (MPM specific)
> ##
>
> # prefork MPM
> # StartServers ......... number of server processes to start
> # MinSpareServers ...... minimum number of server processes which are
> kept spare
> # MaxSpareServers ...... maximum number of server processes which are
> kept spare
> # MaxClients ........... maximum number of server processes allowed to
> start
> # MaxRequestsPerChild .. maximum number of requests a server process
> serves
> <IfModule prefork.c>
> StartServers 5
> MinSpareServers 5
> MaxSpareServers 10
> MaxClients 20
> MaxRequestsPerChild 0
> </IfModule>
>
> # pthread MPM
> # StartServers ......... initial number of server processes to start
> # MaxClients ........... maximum number of server processes allowed to
> start
> # MinSpareThreads ...... minimum number of worker threads which are
> kept spare
> # MaxSpareThreads ...... maximum number of worker threads which are
> kept spare
> # ThreadsPerChild ...... constant number of worker threads in each
> server process
> # MaxRequestsPerChild .. maximum number of requests a server process
> serves
> <IfModule worker.c>
> StartServers 2
> MaxClients 150
> MinSpareThreads 25
> MaxSpareThreads 75
> ThreadsPerChild 25
> MaxRequestsPerChild 0
> </IfModule>
>
> --- snip ---
> So, those are examples. The docs are at this URL:
> http://httpd.apache.org/docs/2.2/mod/mpm_common.html#serverlimit
>
> But you may also want to look for your timeout directives to decrease
> how long those threads sit waiting. Also, the KeepAlive settings may be
> fouling up somehow, and worth changing a bit.
> --- snip ---
> Timeout 300
>
> KeepAlive On
>
> MaxKeepAliveRequests 100
>
> KeepAliveTimeout 15
> --- snip ---
>
> The URL for that stuff is in the Core directive docs.
> http://httpd.apache.org/docs/2.2/mod/core.html#timeout
>
> http://httpd.apache.org/docs/2.2/mod/core.html#keepalive
>
> http://httpd.apache.org/docs/2.2/mod/core.html#maxkeepaliverequests
>
> http://httpd.apache.org/docs/2.2/mod/core.html#keepalivetimeout
>
>
> Hopefully something here could help tune Apache to handle these requests
> better. I can't help much more than you know on blocking the requests,
> however. I've never seen log entries like yours where there is no IP
> address recorded for the connection. That seems pretty odd to me.
>
> -Nick
>
> -----Original Message-----
> From: Vermont Area Group of Unix Enthusiasts [mailto:[EMAIL PROTECTED]
> On Behalf Of Nick Floersch
> Sent: Thursday, May 08, 2008 1:37 PM
> To: [email protected]
> Subject: Re: Apache Problem
>
> Have you ruled out a deliberate DOS attack?
>
> How quickly does it shoot to 255 processes? There is an option to
> increase the number of child threads the master can split off... but I
> suspect that increasing the limit would just provide more room for it to
> fill up. There might be some way to limit the number of connections per
> client. Still, the Apache worker thread controls might be helpful to
> read about here.
>
> Are all of the connections from the same client IP? Or are you getting
> actions from lots of IPs where the child threads don't quit after
> finishing the transactions?
>
> There is a program called Apache Top, sort of like regular UNIX Top...
> but maybe it would help monitor this stuff? Or is that you are using?
>
> Are these threads initiating processes which interact with a database at
> all? Could the database connections not be closing cleanly, leaving
> Apache thinking it is still reading/writing/connected?
>
> What server side scripting are you using? PHP, Perl, Python, Ruby?
>
> What do the logs say?
>
> -Nick
>
> -----Original Message-----
> From: Vermont Area Group of Unix Enthusiasts [mailto:[EMAIL PROTECTED]
> On Behalf Of Phil Marshall
> Sent: Thursday, May 08, 2008 1:23 PM
> To: [email protected]
> Subject: Apache Problem
>
> Here is a sample from the apache status output. We are getting
> flooded with these. Apache will get filled up to 255 connections with
> these things. This has happened before, but usually it goes away
> quickly. Today it went on for about 45 minutes. Its fine now. Does
> anyone have any idea what this is or how to block it?
>
> 51-9 28349 0/1/1 R 0.00 28 3 0.0 0.02
> 0.02 ? ? ..reading..
> 52-9 28350 0/1/1 R 0.00 28 0 0.0 0.000
> 0.000 ? ? ..reading..
> 53-9 28351 0/2/2 R 0.00 28 0 0.0 0.000
> 0.000 ? ? ..reading..
> 54-9 28352 0/1/1 R 0.00 28 0 0.0 0.000
> 0.000 ? ? ..reading..
> 55-9 28354 0/1/1 R 0.00 28 78 0.0 0.00
> 0.00 ? ? ..reading..
> 56-9 28355 0/1/1 R 0.00 28 0 0.0 0.000
> 0.000 ? ? ..reading..
>
> ---------------------
> Phil Marshall
> Signal Advertising
> 535 Stone Cutters Way
> Montpelier, VT 05602
> 802-229-4149
> [EMAIL PROTECTED]
