Re: [CentOS] Apache Issue on CentOS 6
Anyone got any idea? On Mon, Apr 8, 2013 at 11:14 AM, linuxsupport lin.supp...@gmail.com wrote: I was not able to reproduce it while sending so many requests from ab or any other tool, it only appears when requests come from browser, I had posted this question to Apache users as well and someone told that it is due to Chrome uses preconnection, though this feature was available on Chrome since version 7, till kernel version 2.6.31.14 these preconnection requests were not coming to Apache status page but from 2.6.32 they started appearing on Apache status. Following thread also talks about preconnection https://code.google.com/p/chromium/issues/detail?id=85229 On Mon, Apr 8, 2013 at 10:37 AM, Banyan He ban...@rootong.com wrote: Yes, they do because I'm using slow access to attack my servers. To your environment, you can use tcpdump to capture one connection to check if it's the slow access attack. If it's an attack, we focus on fixing that part. If it's the code problem, then, we can get back to the httpd daemon checking what it goes wrong. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/8/2013 1:03 PM, linuxsupport wrote: your both el5 and el6 Apache status show lots of R -- Reading On Mon, Apr 8, 2013 at 10:24 AM, Banyan He ban...@rootong.com wrote: I did a quick test on el5 and el6 with these package, httpd-2.2.3-43.el5.centos httpd-2.2.15-15.el6.centos.1.i686 I kept the configuration as what it is in default. The index page is about 7k, 100 connections per second. I barely find the connection is marked as R. Mostly C and _. This is done by ab from httpd. I also did a quick test with slow attack. It's basically slowing the client itself to collect the data from the server. I did 200 connections per second. My server is ok seems. A little bit slow, but not too much. el5 CWS. el6 RRRCCCCRWCWCCWWCCC.. I also did the capture on the network traffic that I can find out the connections are doing something bad. You may follow the lead here as I mentioned. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/7/2013 12:23 AM, linuxsupport wrote: There is no problem with the hardware, If I installed CentOS 5 then it works well, at a time out of total 44 concurrent requests 34 were in reading state On Sat, Apr 6, 2013 at 2:03 PM, Banyan He ban...@rootong.com wrote: I went to the source code to check this. Seems like it's used for against the slow request attack from the rate. There is a timeout and rate set for header and body. I'd keep that thought, capture one connection from tcpdump seeing if they are doing something bad. If not, you seem need a new server balancing the traffic. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/6/2013 3:06 PM, linuxsupport wrote: I have already checked but all requests are from different IP's and even different subnet When there are less requests it works ok even if there are more than 60% reading requests but during peak time when concurrent requests goes beyond 150, due to reading requests it becomes 300+ requests processing at the same time and that then Apache stop responding as maxclient is set to 300. CPU load also goes up and thing become very slow. On Sat, Apr 6, 2013 at 10:33 AM, Banyan He ban...@rootong.com wrote: I'd recommend you to sort out the connections. Find out if they are coming from the same client or the same subnet of the clients. Doing a simple tcpdump capture to analyze the data seeing if it's a good R or a bad R. Don't really think it's because of the version. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/6/2013 12:24 PM, linuxsupport wrote: I am facing a problem with Apache on CentOS 6 Apache 2.2.19 is complied from source. I see so many reading requests in Apache status page, as per my previous experience this reading request issue mainly comes when any of the internet route having any problem and it request takes time to completely reach to Apache, but this time there is no network issue. I have ran same setup on CentOS 5 it works well, but on CentOS 6 it show 60%+ reading requests, web site has 20-25 requests per second that becomes 80+ I also tried to upgrade Apache to 2.2.24 but it is same on new version as well. Anyone else has experienced this issue?
Re: [CentOS] Apache Issue on CentOS 6
I did a quick test on el5 and el6 with these package, httpd-2.2.3-43.el5.centos httpd-2.2.15-15.el6.centos.1.i686 I kept the configuration as what it is in default. The index page is about 7k, 100 connections per second. I barely find the connection is marked as R. Mostly C and _. This is done by ab from httpd. I also did a quick test with slow attack. It's basically slowing the client itself to collect the data from the server. I did 200 connections per second. My server is ok seems. A little bit slow, but not too much. el5 CWS. el6 RRRCCCCRWCWCCWWCCC.. I also did the capture on the network traffic that I can find out the connections are doing something bad. You may follow the lead here as I mentioned. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/7/2013 12:23 AM, linuxsupport wrote: There is no problem with the hardware, If I installed CentOS 5 then it works well, at a time out of total 44 concurrent requests 34 were in reading state On Sat, Apr 6, 2013 at 2:03 PM, Banyan He ban...@rootong.com mailto:ban...@rootong.com wrote: I went to the source code to check this. Seems like it's used for against the slow request attack from the rate. There is a timeout and rate set for header and body. I'd keep that thought, capture one connection from tcpdump seeing if they are doing something bad. If not, you seem need a new server balancing the traffic. Banyan He Blog:http://www.rootong.com Email:ban...@rootong.com mailto:ban...@rootong.com On 4/6/2013 3:06 PM, linuxsupport wrote: I have already checked but all requests are from different IP's and even different subnet When there are less requests it works ok even if there are more than 60% reading requests but during peak time when concurrent requests goes beyond 150, due to reading requests it becomes 300+ requests processing at the same time and that then Apache stop responding as maxclient is set to 300. CPU load also goes up and thing become very slow. On Sat, Apr 6, 2013 at 10:33 AM, Banyan He ban...@rootong.com mailto:ban...@rootong.com wrote: I'd recommend you to sort out the connections. Find out if they are coming from the same client or the same subnet of the clients. Doing a simple tcpdump capture to analyze the data seeing if it's a good R or a bad R. Don't really think it's because of the version. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com mailto:ban...@rootong.com On 4/6/2013 12:24 PM, linuxsupport wrote: I am facing a problem with Apache on CentOS 6 Apache 2.2.19 is complied from source. I see so many reading requests in Apache status page, as per my previous experience this reading request issue mainly comes when any of the internet route having any problem and it request takes time to completely reach to Apache, but this time there is no network issue. I have ran same setup on CentOS 5 it works well, but on CentOS 6 it show 60%+ reading requests, web site has 20-25 requests per second that becomes 80+ I also tried to upgrade Apache to 2.2.24 but it is same on new version as well. Anyone else has experienced this issue? ___ CentOS mailing list CentOS@centos.org mailto:CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Apache Issue on CentOS 6
your both el5 and el6 Apache status show lots of R -- Reading On Mon, Apr 8, 2013 at 10:24 AM, Banyan He ban...@rootong.com wrote: I did a quick test on el5 and el6 with these package, httpd-2.2.3-43.el5.centos httpd-2.2.15-15.el6.centos.1.i686 I kept the configuration as what it is in default. The index page is about 7k, 100 connections per second. I barely find the connection is marked as R. Mostly C and _. This is done by ab from httpd. I also did a quick test with slow attack. It's basically slowing the client itself to collect the data from the server. I did 200 connections per second. My server is ok seems. A little bit slow, but not too much. el5 CWS. el6 RRRCCCCRWCWCCWWCCC.. I also did the capture on the network traffic that I can find out the connections are doing something bad. You may follow the lead here as I mentioned. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/7/2013 12:23 AM, linuxsupport wrote: There is no problem with the hardware, If I installed CentOS 5 then it works well, at a time out of total 44 concurrent requests 34 were in reading state On Sat, Apr 6, 2013 at 2:03 PM, Banyan He ban...@rootong.com wrote: I went to the source code to check this. Seems like it's used for against the slow request attack from the rate. There is a timeout and rate set for header and body. I'd keep that thought, capture one connection from tcpdump seeing if they are doing something bad. If not, you seem need a new server balancing the traffic. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/6/2013 3:06 PM, linuxsupport wrote: I have already checked but all requests are from different IP's and even different subnet When there are less requests it works ok even if there are more than 60% reading requests but during peak time when concurrent requests goes beyond 150, due to reading requests it becomes 300+ requests processing at the same time and that then Apache stop responding as maxclient is set to 300. CPU load also goes up and thing become very slow. On Sat, Apr 6, 2013 at 10:33 AM, Banyan He ban...@rootong.com wrote: I'd recommend you to sort out the connections. Find out if they are coming from the same client or the same subnet of the clients. Doing a simple tcpdump capture to analyze the data seeing if it's a good R or a bad R. Don't really think it's because of the version. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/6/2013 12:24 PM, linuxsupport wrote: I am facing a problem with Apache on CentOS 6 Apache 2.2.19 is complied from source. I see so many reading requests in Apache status page, as per my previous experience this reading request issue mainly comes when any of the internet route having any problem and it request takes time to completely reach to Apache, but this time there is no network issue. I have ran same setup on CentOS 5 it works well, but on CentOS 6 it show 60%+ reading requests, web site has 20-25 requests per second that becomes 80+ I also tried to upgrade Apache to 2.2.24 but it is same on new version as well. Anyone else has experienced this issue? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Apache Issue on CentOS 6
Yes, they do because I'm using slow access to attack my servers. To your environment, you can use tcpdump to capture one connection to check if it's the slow access attack. If it's an attack, we focus on fixing that part. If it's the code problem, then, we can get back to the httpd daemon checking what it goes wrong. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/8/2013 1:03 PM, linuxsupport wrote: your both el5 and el6 Apache status show lots of R -- Reading On Mon, Apr 8, 2013 at 10:24 AM, Banyan He ban...@rootong.com mailto:ban...@rootong.com wrote: I did a quick test on el5 and el6 with these package, httpd-2.2.3-43.el5.centos httpd-2.2.15-15.el6.centos.1.i686 I kept the configuration as what it is in default. The index page is about 7k, 100 connections per second. I barely find the connection is marked as R. Mostly C and _. This is done by ab from httpd. I also did a quick test with slow attack. It's basically slowing the client itself to collect the data from the server. I did 200 connections per second. My server is ok seems. A little bit slow, but not too much. el5 CWS. el6 RRRCCCCRWCWCCWWCCC.. I also did the capture on the network traffic that I can find out the connections are doing something bad. You may follow the lead here as I mentioned. Banyan He Blog:http://www.rootong.com Email:ban...@rootong.com mailto:ban...@rootong.com On 4/7/2013 12:23 AM, linuxsupport wrote: There is no problem with the hardware, If I installed CentOS 5 then it works well, at a time out of total 44 concurrent requests 34 were in reading state On Sat, Apr 6, 2013 at 2:03 PM, Banyan He ban...@rootong.com mailto:ban...@rootong.com wrote: I went to the source code to check this. Seems like it's used for against the slow request attack from the rate. There is a timeout and rate set for header and body. I'd keep that thought, capture one connection from tcpdump seeing if they are doing something bad. If not, you seem need a new server balancing the traffic. Banyan He Blog:http://www.rootong.com Email:ban...@rootong.com mailto:ban...@rootong.com On 4/6/2013 3:06 PM, linuxsupport wrote: I have already checked but all requests are from different IP's and even different subnet When there are less requests it works ok even if there are more than 60% reading requests but during peak time when concurrent requests goes beyond 150, due to reading requests it becomes 300+ requests processing at the same time and that then Apache stop responding as maxclient is set to 300. CPU load also goes up and thing become very slow. On Sat, Apr 6, 2013 at 10:33 AM, Banyan He ban...@rootong.com mailto:ban...@rootong.com wrote: I'd recommend you to sort out the connections. Find out if they are coming from the same client or the same subnet of the clients. Doing a simple tcpdump capture to analyze the data seeing if it's a good R or a bad R. Don't really think it's because of the version. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com mailto:ban...@rootong.com On 4/6/2013 12:24 PM, linuxsupport wrote: I am facing a problem with Apache on CentOS 6 Apache 2.2.19 is complied from source. I see so many reading requests in Apache status page, as per my previous experience this reading request issue mainly comes when any of the internet route having any problem and it request takes time to completely reach to Apache, but this time there is no network issue. I have ran same setup on CentOS 5 it works well, but on CentOS 6 it show 60%+ reading requests, web site has 20-25 requests per second that becomes 80+ I also tried to upgrade Apache to 2.2.24 but it is same on new
Re: [CentOS] Apache Issue on CentOS 6
I was not able to reproduce it while sending so many requests from ab or any other tool, it only appears when requests come from browser, I had posted this question to Apache users as well and someone told that it is due to Chrome uses preconnection, though this feature was available on Chrome since version 7, till kernel version 2.6.31.14 these preconnection requests were not coming to Apache status page but from 2.6.32 they started appearing on Apache status. Following thread also talks about preconnection https://code.google.com/p/chromium/issues/detail?id=85229 On Mon, Apr 8, 2013 at 10:37 AM, Banyan He ban...@rootong.com wrote: Yes, they do because I'm using slow access to attack my servers. To your environment, you can use tcpdump to capture one connection to check if it's the slow access attack. If it's an attack, we focus on fixing that part. If it's the code problem, then, we can get back to the httpd daemon checking what it goes wrong. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/8/2013 1:03 PM, linuxsupport wrote: your both el5 and el6 Apache status show lots of R -- Reading On Mon, Apr 8, 2013 at 10:24 AM, Banyan He ban...@rootong.com wrote: I did a quick test on el5 and el6 with these package, httpd-2.2.3-43.el5.centos httpd-2.2.15-15.el6.centos.1.i686 I kept the configuration as what it is in default. The index page is about 7k, 100 connections per second. I barely find the connection is marked as R. Mostly C and _. This is done by ab from httpd. I also did a quick test with slow attack. It's basically slowing the client itself to collect the data from the server. I did 200 connections per second. My server is ok seems. A little bit slow, but not too much. el5 CWS. el6 RRRCCCCRWCWCCWWCCC.. I also did the capture on the network traffic that I can find out the connections are doing something bad. You may follow the lead here as I mentioned. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/7/2013 12:23 AM, linuxsupport wrote: There is no problem with the hardware, If I installed CentOS 5 then it works well, at a time out of total 44 concurrent requests 34 were in reading state On Sat, Apr 6, 2013 at 2:03 PM, Banyan He ban...@rootong.com wrote: I went to the source code to check this. Seems like it's used for against the slow request attack from the rate. There is a timeout and rate set for header and body. I'd keep that thought, capture one connection from tcpdump seeing if they are doing something bad. If not, you seem need a new server balancing the traffic. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/6/2013 3:06 PM, linuxsupport wrote: I have already checked but all requests are from different IP's and even different subnet When there are less requests it works ok even if there are more than 60% reading requests but during peak time when concurrent requests goes beyond 150, due to reading requests it becomes 300+ requests processing at the same time and that then Apache stop responding as maxclient is set to 300. CPU load also goes up and thing become very slow. On Sat, Apr 6, 2013 at 10:33 AM, Banyan He ban...@rootong.com wrote: I'd recommend you to sort out the connections. Find out if they are coming from the same client or the same subnet of the clients. Doing a simple tcpdump capture to analyze the data seeing if it's a good R or a bad R. Don't really think it's because of the version. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/6/2013 12:24 PM, linuxsupport wrote: I am facing a problem with Apache on CentOS 6 Apache 2.2.19 is complied from source. I see so many reading requests in Apache status page, as per my previous experience this reading request issue mainly comes when any of the internet route having any problem and it request takes time to completely reach to Apache, but this time there is no network issue. I have ran same setup on CentOS 5 it works well, but on CentOS 6 it show 60%+ reading requests, web site has 20-25 requests per second that becomes 80+ I also tried to upgrade Apache to 2.2.24 but it is same on new version as well. Anyone else has experienced this issue? ___ CentOS mailing list CentOS@centos.org
Re: [CentOS] Apache Issue on CentOS 6
I have already checked but all requests are from different IP's and even different subnet When there are less requests it works ok even if there are more than 60% reading requests but during peak time when concurrent requests goes beyond 150, due to reading requests it becomes 300+ requests processing at the same time and that then Apache stop responding as maxclient is set to 300. CPU load also goes up and thing become very slow. On Sat, Apr 6, 2013 at 10:33 AM, Banyan He ban...@rootong.com wrote: I'd recommend you to sort out the connections. Find out if they are coming from the same client or the same subnet of the clients. Doing a simple tcpdump capture to analyze the data seeing if it's a good R or a bad R. Don't really think it's because of the version. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/6/2013 12:24 PM, linuxsupport wrote: I am facing a problem with Apache on CentOS 6 Apache 2.2.19 is complied from source. I see so many reading requests in Apache status page, as per my previous experience this reading request issue mainly comes when any of the internet route having any problem and it request takes time to completely reach to Apache, but this time there is no network issue. I have ran same setup on CentOS 5 it works well, but on CentOS 6 it show 60%+ reading requests, web site has 20-25 requests per second that becomes 80+ I also tried to upgrade Apache to 2.2.24 but it is same on new version as well. Anyone else has experienced this issue? __**_ CentOS mailing list CentOS@centos.org http://lists.centos.org/**mailman/listinfo/centoshttp://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Apache Issue on CentOS 6
Did you check mpm settings of httpd-mpm.conf ? default response client value for 2.2... versions is 150 and 2 proccessor for simultaneously response applying httpd-mpm.conf in httpd.conf and changing mpm thread settings will help you... On 04/06/2013 10:06 AM, linuxsupport wrote: I have already checked but all requests are from different IP's and even different subnet When there are less requests it works ok even if there are more than 60% reading requests but during peak time when concurrent requests goes beyond 150, due to reading requests it becomes 300+ requests processing at the same time and that then Apache stop responding as maxclient is set to 300. CPU load also goes up and thing become very slow. On Sat, Apr 6, 2013 at 10:33 AM, Banyan He ban...@rootong.com wrote: I'd recommend you to sort out the connections. Find out if they are coming from the same client or the same subnet of the clients. Doing a simple tcpdump capture to analyze the data seeing if it's a good R or a bad R. Don't really think it's because of the version. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/6/2013 12:24 PM, linuxsupport wrote: I am facing a problem with Apache on CentOS 6 Apache 2.2.19 is complied from source. I see so many reading requests in Apache status page, as per my previous experience this reading request issue mainly comes when any of the internet route having any problem and it request takes time to completely reach to Apache, but this time there is no network issue. I have ran same setup on CentOS 5 it works well, but on CentOS 6 it show 60%+ reading requests, web site has 20-25 requests per second that becomes 80+ I also tried to upgrade Apache to 2.2.24 but it is same on new version as well. Anyone else has experienced this issue? __**_ CentOS mailing list CentOS@centos.org http://lists.centos.org/**mailman/listinfo/centoshttp://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Apache Issue on CentOS 6
I'd recommend you to sort out the connections. Find out if they are coming from the same client or the same subnet of the clients. Doing a simple tcpdump capture to analyze the data seeing if it's a good R or a bad R. Don't really think it's because of the version. Banyan He Blog: http://www.rootong.com Email: ban...@rootong.com On 4/6/2013 12:24 PM, linuxsupport wrote: I am facing a problem with Apache on CentOS 6 Apache 2.2.19 is complied from source. I see so many reading requests in Apache status page, as per my previous experience this reading request issue mainly comes when any of the internet route having any problem and it request takes time to completely reach to Apache, but this time there is no network issue. I have ran same setup on CentOS 5 it works well, but on CentOS 6 it show 60%+ reading requests, web site has 20-25 requests per second that becomes 80+ I also tried to upgrade Apache to 2.2.24 but it is same on new version as well. Anyone else has experienced this issue? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
