[users@httpd] Re: apache service interruption
> My server has 4GB RAM and uses nginx as a reverse proxy to apache. A > little while ago my website became inaccessible for about 30 minutes. > I checked my munin graphs and it looks like apache processes spiked to > about 29 during this time which is many times greater than usual. I > have MaxClients at 30 and the error log verifies that MaxClients was > not reached. The strange part is system disk latency shows a spike > during the interruption which is only very slightly greater than other > spikes which did not interrupt service. System CPU, memory, and swap > usage don't show anything interesting at all. > > Does this make sense to anyone? Should I decrease MaxClients? > > - Grant I've looked over my access_log and I can see there is a particular IP which was making many requests during the interruption. Since munin does not show there was an excessive amount of memory or CPU usage, lowering MaxClients won't help? - Grant - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
Was it just an IP exhausting the apache service with too many connections? What do you see in the access logs? I use OSSEC HIDS on my apache servers to mitigate this. -- Sent from my mobile device Michael D. Wood www.itsecuritypros.org Grant wrote: >> My server has 4GB RAM and uses nginx as a reverse proxy to apache. A >> little while ago my website became inaccessible for about 30 minutes. >> I checked my munin graphs and it looks like apache processes spiked to >> about 29 during this time which is many times greater than usual. I >> have MaxClients at 30 and the error log verifies that MaxClients was >> not reached. The strange part is system disk latency shows a spike >> during the interruption which is only very slightly greater than other >> spikes which did not interrupt service. System CPU, memory, and swap >> usage don't show anything interesting at all. >> >> Does this make sense to anyone? Should I decrease MaxClients? >> >> - Grant > >I've looked over my access_log and I can see there is a particular IP >which was making many requests during the interruption. Since munin >does not show there was an excessive amount of memory or CPU usage, >lowering MaxClients won't help? > >- Grant > >- >To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >For additional commands, e-mail: users-h...@httpd.apache.org > - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
> Was it just an IP exhausting the apache service with too many connections? > What do you see in the access logs? I use OSSEC HIDS on my apache servers to > mitigate this. In the access log I see the same IP made many requests during the service interruption and I think that exhausted the apache service. It looks like there isn't a Gentoo ebuild for OSSEC HIDS. Is there another way to prevent this sort of thing? - Grant >>> My server has 4GB RAM and uses nginx as a reverse proxy to apache. A >>> little while ago my website became inaccessible for about 30 minutes. >>> I checked my munin graphs and it looks like apache processes spiked to >>> about 29 during this time which is many times greater than usual. I >>> have MaxClients at 30 and the error log verifies that MaxClients was >>> not reached. The strange part is system disk latency shows a spike >>> during the interruption which is only very slightly greater than other >>> spikes which did not interrupt service. System CPU, memory, and swap >>> usage don't show anything interesting at all. >>> >>> Does this make sense to anyone? Should I decrease MaxClients? >>> >>> - Grant >> >>I've looked over my access_log and I can see there is a particular IP >>which was making many requests during the interruption. Since munin >>does not show there was an excessive amount of memory or CPU usage, >>lowering MaxClients won't help? >> >>- Grant - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
You can always compile from source ;) What version of Apache are you running? On 07/29/2013 02:59 AM, Grant wrote: Was it just an IP exhausting the apache service with too many connections? What do you see in the access logs? I use OSSEC HIDS on my apache servers to mitigate this. In the access log I see the same IP made many requests during the service interruption and I think that exhausted the apache service. It looks like there isn't a Gentoo ebuild for OSSEC HIDS. Is there another way to prevent this sort of thing? - Grant My server has 4GB RAM and uses nginx as a reverse proxy to apache. A little while ago my website became inaccessible for about 30 minutes. I checked my munin graphs and it looks like apache processes spiked to about 29 during this time which is many times greater than usual. I have MaxClients at 30 and the error log verifies that MaxClients was not reached. The strange part is system disk latency shows a spike during the interruption which is only very slightly greater than other spikes which did not interrupt service. System CPU, memory, and swap usage don't show anything interesting at all. Does this make sense to anyone? Should I decrease MaxClients? - Grant I've looked over my access_log and I can see there is a particular IP which was making many requests during the interruption. Since munin does not show there was an excessive amount of memory or CPU usage, lowering MaxClients won't help? - Grant - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
Also, you should be able to limit simultaneous client connections with your firewall and pass the traffic in a syn proxy state. There are numerous ways to achieve this. On 07/29/2013 03:18 AM, Michael D. Wood wrote: You can always compile from source ;) What version of Apache are you running? On 07/29/2013 02:59 AM, Grant wrote: Was it just an IP exhausting the apache service with too many connections? What do you see in the access logs? I use OSSEC HIDS on my apache servers to mitigate this. In the access log I see the same IP made many requests during the service interruption and I think that exhausted the apache service. It looks like there isn't a Gentoo ebuild for OSSEC HIDS. Is there another way to prevent this sort of thing? - Grant My server has 4GB RAM and uses nginx as a reverse proxy to apache. A little while ago my website became inaccessible for about 30 minutes. I checked my munin graphs and it looks like apache processes spiked to about 29 during this time which is many times greater than usual. I have MaxClients at 30 and the error log verifies that MaxClients was not reached. The strange part is system disk latency shows a spike during the interruption which is only very slightly greater than other spikes which did not interrupt service. System CPU, memory, and swap usage don't show anything interesting at all. Does this make sense to anyone? Should I decrease MaxClients? - Grant I've looked over my access_log and I can see there is a particular IP which was making many requests during the interruption. Since munin does not show there was an excessive amount of memory or CPU usage, lowering MaxClients won't help? - Grant - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
> You can always compile from source ;) > What version of Apache are you running? I'm running 2.2.25. - Grant >>> Was it just an IP exhausting the apache service with too many >>> connections? What do you see in the access logs? I use OSSEC HIDS on my >>> apache servers to mitigate this. >> >> >> In the access log I see the same IP made many requests during the >> service interruption and I think that exhausted the apache service. >> It looks like there isn't a Gentoo ebuild for OSSEC HIDS. Is there >> another way to prevent this sort of thing? >> >> - Grant >> >> > My server has 4GB RAM and uses nginx as a reverse proxy to apache. A > little while ago my website became inaccessible for about 30 minutes. > I checked my munin graphs and it looks like apache processes spiked to > about 29 during this time which is many times greater than usual. I > have MaxClients at 30 and the error log verifies that MaxClients was > not reached. The strange part is system disk latency shows a spike > during the interruption which is only very slightly greater than other > spikes which did not interrupt service. System CPU, memory, and swap > usage don't show anything interesting at all. > > Does this make sense to anyone? Should I decrease MaxClients? > > - Grant I've looked over my access_log and I can see there is a particular IP which was making many requests during the interruption. Since munin does not show there was an excessive amount of memory or CPU usage, lowering MaxClients won't help? - Grant >> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >> For additional commands, e-mail: users-h...@httpd.apache.org > > > > - > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > For additional commands, e-mail: users-h...@httpd.apache.org > - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
> Also, you should be able to limit simultaneous client connections with your > firewall and pass the traffic in a syn proxy state. There are numerous ways > to achieve this. Is that the best way to go besides OSSEC HIDS? I can imagine that sort of thing could cause problems. - Grant >> You can always compile from source ;) >> What version of Apache are you running? >> >> On 07/29/2013 02:59 AM, Grant wrote: Was it just an IP exhausting the apache service with too many connections? What do you see in the access logs? I use OSSEC HIDS on my apache servers to mitigate this. >>> >>> >>> In the access log I see the same IP made many requests during the >>> service interruption and I think that exhausted the apache service. >>> It looks like there isn't a Gentoo ebuild for OSSEC HIDS. Is there >>> another way to prevent this sort of thing? >>> >>> - Grant >>> >>> >> My server has 4GB RAM and uses nginx as a reverse proxy to apache. A >> little while ago my website became inaccessible for about 30 minutes. >> I checked my munin graphs and it looks like apache processes spiked to >> about 29 during this time which is many times greater than usual. I >> have MaxClients at 30 and the error log verifies that MaxClients was >> not reached. The strange part is system disk latency shows a spike >> during the interruption which is only very slightly greater than other >> spikes which did not interrupt service. System CPU, memory, and swap >> usage don't show anything interesting at all. >> >> Does this make sense to anyone? Should I decrease MaxClients? >> >> - Grant > > > I've looked over my access_log and I can see there is a particular IP > which was making many requests during the interruption. Since munin > does not show there was an excessive amount of memory or CPU usage, > lowering MaxClients won't help? > > - Grant - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
You wouldn't keep a syn proxy rule enabled all the time; only under a DoS attack. You could also implement ModSecurity. On 07/29/2013 02:07 PM, Grant wrote: Also, you should be able to limit simultaneous client connections with your firewall and pass the traffic in a syn proxy state. There are numerous ways to achieve this. Is that the best way to go besides OSSEC HIDS? I can imagine that sort of thing could cause problems. - Grant You can always compile from source ;) What version of Apache are you running? On 07/29/2013 02:59 AM, Grant wrote: Was it just an IP exhausting the apache service with too many connections? What do you see in the access logs? I use OSSEC HIDS on my apache servers to mitigate this. In the access log I see the same IP made many requests during the service interruption and I think that exhausted the apache service. It looks like there isn't a Gentoo ebuild for OSSEC HIDS. Is there another way to prevent this sort of thing? - Grant My server has 4GB RAM and uses nginx as a reverse proxy to apache. A little while ago my website became inaccessible for about 30 minutes. I checked my munin graphs and it looks like apache processes spiked to about 29 during this time which is many times greater than usual. I have MaxClients at 30 and the error log verifies that MaxClients was not reached. The strange part is system disk latency shows a spike during the interruption which is only very slightly greater than other spikes which did not interrupt service. System CPU, memory, and swap usage don't show anything interesting at all. Does this make sense to anyone? Should I decrease MaxClients? - Grant I've looked over my access_log and I can see there is a particular IP which was making many requests during the interruption. Since munin does not show there was an excessive amount of memory or CPU usage, lowering MaxClients won't help? - Grant - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
> You wouldn't keep a syn proxy rule enabled all the time; only under a DoS > attack. You could also implement ModSecurity. ModSecurity looks good and I think it works with nginx as well as apache. Is everyone who isn't running OSSEC HIDS or ModSecurity vulnerable to a single client requesting too many pages and interrupting the service? - Grant >>> Also, you should be able to limit simultaneous client connections with >>> your >>> firewall and pass the traffic in a syn proxy state. There are numerous >>> ways >>> to achieve this. >> >> >> Is that the best way to go besides OSSEC HIDS? I can imagine that >> sort of thing could cause problems. >> >> - Grant >> >> You can always compile from source ;) What version of Apache are you running? On 07/29/2013 02:59 AM, Grant wrote: >> >> >> Was it just an IP exhausting the apache service with too many >> connections? What do you see in the access logs? I use OSSEC HIDS on >> my >> apache servers to mitigate this. > > > > In the access log I see the same IP made many requests during the > service interruption and I think that exhausted the apache service. > It looks like there isn't a Gentoo ebuild for OSSEC HIDS. Is there > another way to prevent this sort of thing? > > - Grant > > My server has 4GB RAM and uses nginx as a reverse proxy to apache. A little while ago my website became inaccessible for about 30 minutes. I checked my munin graphs and it looks like apache processes spiked to about 29 during this time which is many times greater than usual. I have MaxClients at 30 and the error log verifies that MaxClients was not reached. The strange part is system disk latency shows a spike during the interruption which is only very slightly greater than other spikes which did not interrupt service. System CPU, memory, and swap usage don't show anything interesting at all. Does this make sense to anyone? Should I decrease MaxClients? - Grant >>> >>> >>> >>> I've looked over my access_log and I can see there is a particular IP >>> which was making many requests during the interruption. Since munin >>> does not show there was an excessive amount of memory or CPU usage, >>> lowering MaxClients won't help? >>> >>> - Grant - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
Two different things come to mind. Kingcope found an Apache byterange vulnerability and the PoC code he wrote for it exhausts the resources on a server running Apache. Only 1 instance of his perl script had to be ran. LOIC is another that could possible DoS your server from one source. What IP address was hitting your box when this happened? On 07/30/2013 02:25 AM, Grant wrote: You wouldn't keep a syn proxy rule enabled all the time; only under a DoS attack. You could also implement ModSecurity. ModSecurity looks good and I think it works with nginx as well as apache. Is everyone who isn't running OSSEC HIDS or ModSecurity vulnerable to a single client requesting too many pages and interrupting the service? - Grant Also, you should be able to limit simultaneous client connections with your firewall and pass the traffic in a syn proxy state. There are numerous ways to achieve this. Is that the best way to go besides OSSEC HIDS? I can imagine that sort of thing could cause problems. - Grant You can always compile from source ;) What version of Apache are you running? On 07/29/2013 02:59 AM, Grant wrote: Was it just an IP exhausting the apache service with too many connections? What do you see in the access logs? I use OSSEC HIDS on my apache servers to mitigate this. In the access log I see the same IP made many requests during the service interruption and I think that exhausted the apache service. It looks like there isn't a Gentoo ebuild for OSSEC HIDS. Is there another way to prevent this sort of thing? - Grant My server has 4GB RAM and uses nginx as a reverse proxy to apache. A little while ago my website became inaccessible for about 30 minutes. I checked my munin graphs and it looks like apache processes spiked to about 29 during this time which is many times greater than usual. I have MaxClients at 30 and the error log verifies that MaxClients was not reached. The strange part is system disk latency shows a spike during the interruption which is only very slightly greater than other spikes which did not interrupt service. System CPU, memory, and swap usage don't show anything interesting at all. Does this make sense to anyone? Should I decrease MaxClients? - Grant I've looked over my access_log and I can see there is a particular IP which was making many requests during the interruption. Since munin does not show there was an excessive amount of memory or CPU usage, lowering MaxClients won't help? - Grant - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
On Mon, Jul 29, 2013 at 11:25:26PM -0700, Grant wrote: > ModSecurity looks good and I think it works with nginx as well as > apache. Is everyone who isn't running OSSEC HIDS or ModSecurity > vulnerable to a single client requesting too many pages and > interrupting the service? Not everyone, no. There are other alternatives such as mod_limitipconn and mod_reqtimeout to help with such problems as well. Pete -- Openstrike - improving business through open source http://www.openstrike.co.uk/ or call 01722 770036 / 07092 020107 pgpHpyNWw3F78.pgp Description: PGP signature
Re: [users@httpd] Re: apache service interruption
>> ModSecurity looks good and I think it works with nginx as well as >> apache. Is everyone who isn't running OSSEC HIDS or ModSecurity >> vulnerable to a single client requesting too many pages and >> interrupting the service? > > Not everyone, no. There are other alternatives such as mod_limitipconn > and mod_reqtimeout to help with such problems as well. mod_limitipconn sounded like the perfect solution until I started thinking about how many people use the same IP address in some environments like university campuses. I could end up creating a lot more problems than I solve. Does ModSecurity have the same potential downside? Would mod_remoteip prevent this? Is mod_reqtimeout a better solution? I found the following config recommended online within the context of Slowloris attack mitigation: RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500 Will that do anything to prevent someone from opening too many connections and interrupting the apache service? - Grant - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
> Two different things come to mind. Kingcope found an Apache byterange > vulnerability and the PoC code he wrote for it exhausts the resources on a > server running Apache. Only 1 instance of his perl script had to be ran. > LOIC is another that could possible DoS your server from one source. What > IP address was hitting your box when this happened? I'd rather not post the IP if that's OK. I did notice my access_log entries were out of chronological order for the IP address in question. Does that indicate a Slowloris attack? Maybe it's just the result of the server bogging down in response to so many requests in a short amount of time. So I'm sure I understand, a regular browser or unsophisticated script shouldn't be able to interrupt apache service by simply requesting a large number of pages in a short amount of time? If not, how does apache prevent that from happening? - Grant >>> You wouldn't keep a syn proxy rule enabled all the time; only under a DoS >>> attack. You could also implement ModSecurity. >> >> >> ModSecurity looks good and I think it works with nginx as well as >> apache. Is everyone who isn't running OSSEC HIDS or ModSecurity >> vulnerable to a single client requesting too many pages and >> interrupting the service? >> >> - Grant >> >> > Also, you should be able to limit simultaneous client connections with > your > firewall and pass the traffic in a syn proxy state. There are numerous > ways > to achieve this. Is that the best way to go besides OSSEC HIDS? I can imagine that sort of thing could cause problems. - Grant >> You can always compile from source ;) >> What version of Apache are you running? >> >> On 07/29/2013 02:59 AM, Grant wrote: Was it just an IP exhausting the apache service with too many connections? What do you see in the access logs? I use OSSEC HIDS on my apache servers to mitigate this. >>> >>> >>> >>> >>> In the access log I see the same IP made many requests during the >>> service interruption and I think that exhausted the apache service. >>> It looks like there isn't a Gentoo ebuild for OSSEC HIDS. Is there >>> another way to prevent this sort of thing? >>> >>> - Grant >>> >>> >> My server has 4GB RAM and uses nginx as a reverse proxy to apache. >> A >> little while ago my website became inaccessible for about 30 >> minutes. >> I checked my munin graphs and it looks like apache processes >> spiked >> to >> about 29 during this time which is many times greater than usual. >> I >> have MaxClients at 30 and the error log verifies that MaxClients >> was >> not reached. The strange part is system disk latency shows a >> spike >> during the interruption which is only very slightly greater than >> other >> spikes which did not interrupt service. System CPU, memory, and >> swap >> usage don't show anything interesting at all. >> >> Does this make sense to anyone? Should I decrease MaxClients? >> >> - Grant > > > > > I've looked over my access_log and I can see there is a particular > IP > which was making many requests during the interruption. Since > munin > does not show there was an excessive amount of memory or CPU usage, > lowering MaxClients won't help? > > - Grant - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
Truthfully, I've always limited connections from the source IP via a firewall before the traffic is even passed to apache. On 08/01/2013 04:39 AM, Grant wrote: Two different things come to mind. Kingcope found an Apache byterange vulnerability and the PoC code he wrote for it exhausts the resources on a server running Apache. Only 1 instance of his perl script had to be ran. LOIC is another that could possible DoS your server from one source. What IP address was hitting your box when this happened? I'd rather not post the IP if that's OK. I did notice my access_log entries were out of chronological order for the IP address in question. Does that indicate a Slowloris attack? Maybe it's just the result of the server bogging down in response to so many requests in a short amount of time. So I'm sure I understand, a regular browser or unsophisticated script shouldn't be able to interrupt apache service by simply requesting a large number of pages in a short amount of time? If not, how does apache prevent that from happening? - Grant You wouldn't keep a syn proxy rule enabled all the time; only under a DoS attack. You could also implement ModSecurity. ModSecurity looks good and I think it works with nginx as well as apache. Is everyone who isn't running OSSEC HIDS or ModSecurity vulnerable to a single client requesting too many pages and interrupting the service? - Grant Also, you should be able to limit simultaneous client connections with your firewall and pass the traffic in a syn proxy state. There are numerous ways to achieve this. Is that the best way to go besides OSSEC HIDS? I can imagine that sort of thing could cause problems. - Grant You can always compile from source ;) What version of Apache are you running? On 07/29/2013 02:59 AM, Grant wrote: Was it just an IP exhausting the apache service with too many connections? What do you see in the access logs? I use OSSEC HIDS on my apache servers to mitigate this. In the access log I see the same IP made many requests during the service interruption and I think that exhausted the apache service. It looks like there isn't a Gentoo ebuild for OSSEC HIDS. Is there another way to prevent this sort of thing? - Grant My server has 4GB RAM and uses nginx as a reverse proxy to apache. A little while ago my website became inaccessible for about 30 minutes. I checked my munin graphs and it looks like apache processes spiked to about 29 during this time which is many times greater than usual. I have MaxClients at 30 and the error log verifies that MaxClients was not reached. The strange part is system disk latency shows a spike during the interruption which is only very slightly greater than other spikes which did not interrupt service. System CPU, memory, and swap usage don't show anything interesting at all. Does this make sense to anyone? Should I decrease MaxClients? - Grant I've looked over my access_log and I can see there is a particular IP which was making many requests during the interruption. Since munin does not show there was an excessive amount of memory or CPU usage, lowering MaxClients won't help? - Grant - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
> Truthfully, I've always limited connections from the source IP via a > firewall before the traffic is even passed to apache. Do you do this only when under DoS attack or all the time? Won't you potentially prevent legitimate users from making a single connection if they're connecting with a shared IP from a university campus (for example)? How is this accomplished with iptables? - Grant >>> Two different things come to mind. Kingcope found an Apache byterange >>> vulnerability and the PoC code he wrote for it exhausts the resources on >>> a >>> server running Apache. Only 1 instance of his perl script had to be ran. >>> LOIC is another that could possible DoS your server from one source. What >>> IP address was hitting your box when this happened? >> >> >> I'd rather not post the IP if that's OK. I did notice my access_log >> entries were out of chronological order for the IP address in >> question. Does that indicate a Slowloris attack? Maybe it's just the >> result of the server bogging down in response to so many requests in a >> short amount of time. >> >> So I'm sure I understand, a regular browser or unsophisticated script >> shouldn't be able to interrupt apache service by simply requesting a >> large number of pages in a short amount of time? If not, how does >> apache prevent that from happening? >> >> - Grant >> >> > You wouldn't keep a syn proxy rule enabled all the time; only under a > DoS > attack. You could also implement ModSecurity. ModSecurity looks good and I think it works with nginx as well as apache. Is everyone who isn't running OSSEC HIDS or ModSecurity vulnerable to a single client requesting too many pages and interrupting the service? - Grant >>> Also, you should be able to limit simultaneous client connections >>> with >>> your >>> firewall and pass the traffic in a syn proxy state. There are >>> numerous >>> ways >>> to achieve this. >> >> >> >> >> Is that the best way to go besides OSSEC HIDS? I can imagine that >> sort of thing could cause problems. >> >> - Grant >> >> You can always compile from source ;) What version of Apache are you running? On 07/29/2013 02:59 AM, Grant wrote: >> >> >> >> >> Was it just an IP exhausting the apache service with too many >> connections? What do you see in the access logs? I use OSSEC >> HIDS >> on >> my >> apache servers to mitigate this. > > > > > > In the access log I see the same IP made many requests during the > service interruption and I think that exhausted the apache service. > It looks like there isn't a Gentoo ebuild for OSSEC HIDS. Is there > another way to prevent this sort of thing? > > - Grant > > My server has 4GB RAM and uses nginx as a reverse proxy to apache. A little while ago my website became inaccessible for about 30 minutes. I checked my munin graphs and it looks like apache processes spiked to about 29 during this time which is many times greater than usual. I have MaxClients at 30 and the error log verifies that MaxClients was not reached. The strange part is system disk latency shows a spike during the interruption which is only very slightly greater than other spikes which did not interrupt service. System CPU, memory, and swap usage don't show anything interesting at all. Does this make sense to anyone? Should I decrease MaxClients? - Grant >>> >>> >>> >>> >>> >>> I've looked over my access_log and I can see there is a >>> particular >>> IP >>> which was making many requests during the interruption. Since >>> munin >>> does not show there was an excessive amount of memory or CPU >>> usage, >>> lowering MaxClients won't help? >>> >>> - Grant - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
You could potentially deny legitimate users access. I limit so many connections per second per source IP. If I knew I were getting a ton of traffic from a University I would have to adjust it accordingly. The setting in pfsense is Maximum new connections / per second(s) - that's per IP. My site I wouldn't say is pegged with University traffic sharing the same IP. I'm just giving you examples and tailor to your needs. If you get a bunch of traffic from a shared IP, obviously, this would not be the best way to go. I try to mitigate using rate limiting. I don't like to wait for the traffic to pass to Apache and have to configure a module to fix it. Apache should be handling web requests, not having to deal with tons of traffic (bruteforce/DoS). I try to handle that stuff before it even gets passed to Apache. From the Cisco side you could implement ACL's and rate limiting. http://www.debian-administration.org/articles/187 On 08/02/2013 01:49 AM, Grant wrote: Truthfully, I've always limited connections from the source IP via a firewall before the traffic is even passed to apache. Do you do this only when under DoS attack or all the time? Won't you potentially prevent legitimate users from making a single connection if they're connecting with a shared IP from a university campus (for example)? How is this accomplished with iptables? - Grant Two different things come to mind. Kingcope found an Apache byterange vulnerability and the PoC code he wrote for it exhausts the resources on a server running Apache. Only 1 instance of his perl script had to be ran. LOIC is another that could possible DoS your server from one source. What IP address was hitting your box when this happened? I'd rather not post the IP if that's OK. I did notice my access_log entries were out of chronological order for the IP address in question. Does that indicate a Slowloris attack? Maybe it's just the result of the server bogging down in response to so many requests in a short amount of time. So I'm sure I understand, a regular browser or unsophisticated script shouldn't be able to interrupt apache service by simply requesting a large number of pages in a short amount of time? If not, how does apache prevent that from happening? - Grant You wouldn't keep a syn proxy rule enabled all the time; only under a DoS attack. You could also implement ModSecurity. ModSecurity looks good and I think it works with nginx as well as apache. Is everyone who isn't running OSSEC HIDS or ModSecurity vulnerable to a single client requesting too many pages and interrupting the service? - Grant Also, you should be able to limit simultaneous client connections with your firewall and pass the traffic in a syn proxy state. There are numerous ways to achieve this. Is that the best way to go besides OSSEC HIDS? I can imagine that sort of thing could cause problems. - Grant You can always compile from source ;) What version of Apache are you running? On 07/29/2013 02:59 AM, Grant wrote: Was it just an IP exhausting the apache service with too many connections? What do you see in the access logs? I use OSSEC HIDS on my apache servers to mitigate this. In the access log I see the same IP made many requests during the service interruption and I think that exhausted the apache service. It looks like there isn't a Gentoo ebuild for OSSEC HIDS. Is there another way to prevent this sort of thing? - Grant My server has 4GB RAM and uses nginx as a reverse proxy to apache. A little while ago my website became inaccessible for about 30 minutes. I checked my munin graphs and it looks like apache processes spiked to about 29 during this time which is many times greater than usual. I have MaxClients at 30 and the error log verifies that MaxClients was not reached. The strange part is system disk latency shows a spike during the interruption which is only very slightly greater than other spikes which did not interrupt service. System CPU, memory, and swap usage don't show anything interesting at all. Does this make sense to anyone? Should I decrease MaxClients? - Grant I've looked over my access_log and I can see there is a particular IP which was making many requests during the interruption. Since munin does not show there was an excessive amount of memory or CPU usage, lowering MaxClients won't help? - Grant - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Re: apache service interruption
On Thu, Aug 01, 2013 at 10:49:59PM -0700, Grant wrote: > Do you do this only when under DoS attack or all the time? All the time. > Won't you potentially prevent legitimate users from making a single > connection if they're connecting with a shared IP from a university > campus (for example)? Yes. However, if you don't do it you potentially prevent legitimate users from anywhere from making a connection because some greedy user is using up all your server's resources. > How is this accomplished with iptables? With connlimit and/or one of the rate-limiting modules. Just to bring it back on topic, the disadvantage of implementing this at the firewall is that it is very broad-brush (unless you use DPI). You will be limiting connections regardless of the target vhost or path or MIME type or whatever. By doing it in apache with mod_limitipconn or similar you can easily apply stricter limits to heavier content, for example. So, IMHO the best plan is to put an absolute limit in the firewall for the worst possible scenario but then tailor the individual limits for vhosts and content types etc. within apache. Pete -- Openstrike - improving business through open source http://www.openstrike.co.uk/ or call 01722 770036 / 07092 020107 pgp0GpUkERS5h.pgp Description: PGP signature
