Re: Apache Process going high even after restart of backend server tomcat
Hello Andre Warnier, Thanks for the response. 2) tell us what the platform is (O.S.) - it is solaris platform :- uname -a SunOS 5.10 Generic_141414-10 sun4u sparc SUNW,Sun-Fire-V890 3) if you can, also tell what exact version of the mod_jk moduile you are using. You should be able to find this in the message that Apache httpd prints in its error log when it starts. -mod_jk/1.2.28 3) in the configuration of Apache httpd, there must be configuration lines starting with Jk... Copy them here. -LoadModule jk_module modules/mod_jk.so JkWorkersFile /opt/web/apache/app/conf/workers.properties JkLogFile /opt/web/apache/app/logs/mod_jk.log JkLogLevel trace JkLogStampFormat [%a %b %d %H:%M:%S %Y] JkOptions +ForwardURIEscaped +ForwardURICompatUnparsed JkRequestLogFormat %w %V %T JkMount /* worker1 4) copy here the full content of the workers.properties file (comments removed) -workers.tomcat_home=/opt/app/tomcat/5.5.9 workers.java_home=/opt/app/j2sdk1.4.2_02/ worker.list=worker1 worker.worker1.type=ajp13 worker.worker1.host=localhost worker.worker1.port=20209 worker.worker1.socket_keepalive=1 5) in the Tomcat conf/server.xml file, there is a Connector element with a protocol=AJP. Copy this Connector element here. - !-- Define an AJP 1.3 Connector on port 8009 -- Connector port=20209 enableLookups=false redirectPort=8443 protocol=AJP/1.3 / 6) if the platform is Unix/Linux, execute the following command and copy the result here : apache2ctl -V -./apachectl -V Server version: Apache/2.0.59 Server built: Feb 18 2010 09:10:59 Server's Module Magic Number: 20020903:12 Server loaded: APR 0.9.12, APR-UTIL 0.9.12 Compiled using: APR 0.9.12, APR-UTIL 0.9.12 Architecture: 32-bit Server compiled with -D APACHE_MPM_DIR=server/mpm/prefork -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FCNTL_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D HTTPD_ROOT=/opt/web/apache/app/erib2b/ -D SUEXEC_BIN=/opt/web/apache/app/erib2b//bin/suexec -D DEFAULT_PIDLOG=logs/httpd.pid -D DEFAULT_SCOREBOARD=logs/apache_runtime_status -D DEFAULT_LOCKFILE=logs/accept.lock -D DEFAULT_ERRORLOG=logs/error_log -D AP_TYPES_CONFIG_FILE=conf/mime.types -D SERVER_CONFIG_FILE=conf/httpd.conf 7) if the platform is Unix/Linux, go to the bin directory of Tomcat, execute the command version.sh and copy the result here. - Tomcat version is 5.5.9. Please let me know in case i can provide more information. --- On Tue, 1/11/11, André Warnier a...@ice-sa.com wrote: From: André Warnier a...@ice-sa.com Subject: Re: Apache Process going high even after restart of backend server tomcat To: Tomcat Users List users@tomcat.apache.org Date: Tuesday, January 11, 2011, 8:08 PM Amol Puglia wrote: Hello Andre, When i posted same message to apache users list they said the apache process is going high because there is a problem in backend server tomcat and asked me to collect thread dump and post same in tomcat users list. so i posetd thread dump sanpshot in tomcat users list. It seems tomcat is not releasing connection and this is the reason apache proces count is goind high. I would like to know why tomcat is not releasing connection and wanted to figure out whether ther is problem in tomcat or application deployed on tomcat. I have taken therad dump and trying to analyze it but unable to understand what is causing problem by lloking at threads. Let em know if i can attache thread dump here so that experts from the forums can help me to understand thread dump. Allright, so now we may start getting an idea of the problem. Maybe the first things to do : 1) do not post files as attachments in this list. Mostly, the server strips them. Instead, if we ask you for some text information, do a copy-and-paste, and paste the content directly in your email to the list. Also, when you copy contents of files here, please remove the comments, and any private information such as internet hostnames or passwords. 2) tell us what the platform is (O.S.) 3) if you can, also tell what exact version of the mod_jk moduile you are using. You should be able to find this in the message that Apache httpd prints in its error log when it starts. 3) in the configuration of Apache httpd, there must be configuration lines starting with Jk... Copy them here. 4) copy here the full content of the workers.properties file (comments removed) 5) in the Tomcat conf/server.xml file, there is a Connector element with a protocol=AJP. Copy this Connector element here. 6) if the platform is Unix/Linux, execute the following command and copy the result here : apache2ctl -V 7) if the platform is Unix/Linux, go to the bin directory of Tomcat, execute the command version.sh and copy the result here. With the above, we will get a much better idea of your configuration
Re: Apache Process going high even after restart of backend server tomcat
Hello Rainer Jung, Thanks for the response. I can see there are many threads like below which are in sleep state. I have taken thread dumps 4 or 5 times and can see this threads in all the dumps. For the instance i am pasting some of similar kind of threads from the thread dump. I have to show this to developers but before that i have to be very much sure there is no issue from apache web server or tomcat application server end. TP-Processor1 daemon prio=5 tid=0x000a8538 nid=0x24 waiting on condition [9747e000..9747fc30] at java.lang.Thread.sleep(Native Method) at com.ericsson.eai.b2b.MUXServlet.doPost(MUXServlet.java:506) at javax.servlet.http.HttpServlet.service(HttpServlet.java:709) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:307) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:385) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678) at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:534) TP-Processor2 daemon prio=5 tid=0x000a90c0 nid=0x25 waiting on condition [9737e000..9737fc30] at java.lang.Thread.sleep(Native Method) at com.ericsson.eai.b2b.MUXServlet.doPost(MUXServlet.java:506) at javax.servlet.http.HttpServlet.service(HttpServlet.java:709) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:307) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:385) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678) at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:534) --- On Tue, 1/11/11, Rainer Jung rainer.j...@kippdata.de wrote: From: Rainer Jung rainer.j...@kippdata.de Subject: Re: Apache Process going high even after restart of backend server tomcat To: Tomcat Users List users@tomcat.apache.org Date: Tuesday, January 11, 2011, 10:51 PM On 10.01.2011 17:00, Amol Puglia wrote: Hello Team, I am facing very weird problem of apache porcess going high upto 259 and increasing again even after restart of backend server ie. tomcat. Apache version :- 2.0.59 Tomcat Version :- 5.5.9 Apache/Tomcat connectivity :- mod_jk when i have checked mod_jk.log file i am seeing this messages in the logs. [Mon Jan 10 08:14:52 2011] worker1 erib2bdev.ericsson.net 0.001122 [Mon Jan 10 08:14:55 2011] worker1 erib2bdev.ericsson.net 0.000732 [Mon Jan 10 08:14:57 2011] worker1 erib2bdev.ericsson.net 0.000740 [Mon Jan 10 08:15:00 2011] worker1 erib2bdev.ericsson.net 0.000846 [Mon Jan 10 08:15:02 2011] worker1 erib2bdev.ericsson.net 0.000952 As I already wrote on the httpd users list: The lines come from the JkRequestLogFormat configuration setting that you must have added somewhere (default: no such setting, no lines). The configured format will tell you, what the entry means. Likely it is either the name of the virtual host, or the value of the host header the client sent. Cleartext
Re: Apache Process going high even after restart of backend server tomcat
So its time you investigate what this MuxServlet is doing... regards Leon On Wed, Jan 12, 2011 at 9:31 AM, Amol Puglia amolcpug...@yahoo.com wrote: Hello Rainer Jung, Thanks for the response. I can see there are many threads like below which are in sleep state. I have taken thread dumps 4 or 5 times and can see this threads in all the dumps. For the instance i am pasting some of similar kind of threads from the thread dump. I have to show this to developers but before that i have to be very much sure there is no issue from apache web server or tomcat application server end. TP-Processor1 daemon prio=5 tid=0x000a8538 nid=0x24 waiting on condition [9747e000..9747fc30] at java.lang.Thread.sleep(Native Method) at com.ericsson.eai.b2b.MUXServlet.doPost(MUXServlet.java:506) at javax.servlet.http.HttpServlet.service(HttpServlet.java:709) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:307) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:385) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678) at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:534) TP-Processor2 daemon prio=5 tid=0x000a90c0 nid=0x25 waiting on condition [9737e000..9737fc30] at java.lang.Thread.sleep(Native Method) at com.ericsson.eai.b2b.MUXServlet.doPost(MUXServlet.java:506) at javax.servlet.http.HttpServlet.service(HttpServlet.java:709) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:307) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:385) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678) at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:534) --- On Tue, 1/11/11, Rainer Jung rainer.j...@kippdata.de wrote: From: Rainer Jung rainer.j...@kippdata.de Subject: Re: Apache Process going high even after restart of backend server tomcat To: Tomcat Users List users@tomcat.apache.org Date: Tuesday, January 11, 2011, 10:51 PM On 10.01.2011 17:00, Amol Puglia wrote: Hello Team, I am facing very weird problem of apache porcess going high upto 259 and increasing again even after restart of backend server ie. tomcat. Apache version :- 2.0.59 Tomcat Version :- 5.5.9 Apache/Tomcat connectivity :- mod_jk when i have checked mod_jk.log file i am seeing this messages in the logs. [Mon Jan 10 08:14:52 2011] worker1 erib2bdev.ericsson.net 0.001122 [Mon Jan 10 08:14:55 2011] worker1 erib2bdev.ericsson.net 0.000732 [Mon Jan 10 08:14:57 2011] worker1 erib2bdev.ericsson.net 0.000740 [Mon Jan 10 08:15:00 2011] worker1 erib2bdev.ericsson.net 0.000846 [Mon Jan 10 08:15:02 2011] worker1 erib2bdev.ericsson.net 0.000952 As I already wrote on the httpd users list: The lines come from the JkRequestLogFormat configuration setting that you must have
Re: Apache Process going high even after restart of backend server tomcat
Leon Rosenberg wrote: So its time you investigate what this MuxServlet is doing... Yes. And Amol, just to clear another question : erib2bdev.ericsson.net, that /is/ your own server, yes ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Process going high even after restart of backend server tomcat
Hello Andre Warnier, Yes it is DNS name of our one of the server --- On Wed, 1/12/11, André Warnier a...@ice-sa.com wrote: From: André Warnier a...@ice-sa.com Subject: Re: Apache Process going high even after restart of backend server tomcat To: Tomcat Users List users@tomcat.apache.org Date: Wednesday, January 12, 2011, 4:32 PM Leon Rosenberg wrote: So its time you investigate what this MuxServlet is doing... Yes. And Amol, just to clear another question : erib2bdev.ericsson.net, that /is/ your own server, yes ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Process going high even after restart of backend server tomcat
Thank you for the data below. Some additional tips : First : you are running old versions of Apache httpd, mod_jk and Tomcat (and probably also of the Java JVM). You should really update these programs to more recent versions (at least Apache httpd to 2.2.x, and mod_jk to the latest available). If you do not, then - whatever we tell you here may not be relevant for these older versions - you may have problems now, that are due to bugs that have already be corrected in later versions, and then this would all be a big loss of time, for you and for everyone else Maybe the very first thing to do, would be to have a look at your Apache httpd access log. Usually, this is configured so that each line shows a timestamp, the request URL, and the total time that it took to process this request. What does this log tell you ? From the Tomcat thread dump you posted, it looks like there are a lot of threads just sitting there sleeping, instead of being busy processing requests. Are those the threads that should be processing the requests ? And if so, why are they sleeping instead of doing their job ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Process going high even after restart of backend server tomcat
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Amol, On 1/12/2011 3:31 AM, Amol Puglia wrote: I can see there are many threads like below which are in sleep state. I have taken thread dumps 4 or 5 times and can see this threads in all the dumps. For the instance i am pasting some of similar kind of threads from the thread dump. I have to show this to developers but before that i have to be very much sure there is no issue from apache web server or tomcat application server end. TP-Processor1 daemon prio=5 tid=0x000a8538 nid=0x24 waiting on condition [9747e000..9747fc30] at java.lang.Thread.sleep(Native Method) at com.ericsson.eai.b2b.MUXServlet.doPost(MUXServlet.java:506) at javax.servlet.http.HttpServlet.service(HttpServlet.java:709) It's pretty clear that this is not a Tomcat issue: it's time to go to your developers. Wouldn't it have been easier to go to them in the first place? If they had identified a potential problem with Tomcat, it would have saved everyone time. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk0t7joACgkQ9CaO5/Lv0PAmAgCgvd/R4GKb6TANDZYMwz5Zumps C+wAoKlGTshHJo09zB8wusB6xPEmPysB =/BRD -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Process going high even after restart of backend server tomcat
Hello Andre Warnier, Thanks for the response. Is there anyway i can bring down this connections. I would like to know why this process id's are not getting terminated automatically. I have run the traceroute command and following is the output of the command. Please let me know in case any further information is required. traceroute erib2bdev.ericsson.net traceroute to erib2bdev.ericsson.net (193.180.14.127), 30 hops max, 40 byte packets 1 xnet-web-ecn-gw.sw.ericsson.se (153.88.240.1) 0.728 ms 0.611 ms 0.650 ms 2 vip-seebeyond-2.ss.sw.ericsson.se (193.180.14.127) 0.925 ms 0.837 ms 0.859 ms --- On Mon, 1/10/11, André Warnier a...@ice-sa.com wrote: From: André Warnier a...@ice-sa.com Subject: Re: Apache Process going high even after restart of backend server tomcat To: Tomcat Users List users@tomcat.apache.org Date: Monday, January 10, 2011, 9:49 PM Amol Puglia wrote: Hello Team, I am facing very weird problem of apache porcess going high upto 259 and increasing again even after restart of backend server ie. tomcat. Apache version :- 2.0.59 Tomcat Version :- 5.5.9 Apache/Tomcat connectivity :- mod_jk when i have checked mod_jk.log file i am seeing this messages in the logs. [Mon Jan 10 08:14:52 2011] worker1 erib2bdev.ericsson.net 0.001122 [Mon Jan 10 08:14:55 2011] worker1 erib2bdev.ericsson.net 0.000732 [Mon Jan 10 08:14:57 2011] worker1 erib2bdev.ericsson.net 0.000740 [Mon Jan 10 08:15:00 2011] worker1 erib2bdev.ericsson.net 0.000846 [Mon Jan 10 08:15:02 2011] worker1 erib2bdev.ericsson.net 0.000952 I am not sure what does it mean when the name erib2bdev.ericsson.net is not refereed anywhere in my apache and tomcat configuration files and not even in workers.properties file. Can anyone help me torubleshoot this issue? What I can tell you, is that the above is a real system on the Internet. I suggest trying : nslookup erib2bdev.ericsson.net traceroute erib2bdev.ericsson.net Then try http://www.whois.net/whois/ericsson.net and then try to figure out why an application on your Apache server has its own mod_jk configuration, and calls back to this Internet server every 2 seconds or so.. Such as, have you paid your phone bill ? ;-) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Process going high even after restart of backend server tomcat
Amol Puglia wrote: Hello Andre Warnier, Thanks for the response. Is there anyway i can bring down this connections. I would like to know why this process id's are not getting terminated automatically. I have run the traceroute command and following is the output of the command. Hi. After your other post, entitled Need help in understanding tomcat thread dump, it is a bit confusing now what your real problem is. That other post shows that you are runnig some java modules on your tomcat server, which seem to have something to do with the same supplier (Ericsson ?) than the one some Apache processes are connecting to using mod_jk. We have no idea here of your applications, what they are doing, if it is normal or not that they connect to an Internet server of the Ericsson company, and so on. Let's restart from the beginning, your original post : I am facing very weird problem of apache porcess going high upto 259 and increasing again even after restart of backend server ie. tomcat. So, is it the Apache httpd processes (children) that go very high (meaning that there are many, too many ?), or the back-end Tomcat threads ? What is the problem ? is it in Apache httpd, or in Tomcat ? does your server stop responding ? is it slow ? does it not do what it is supposed to do ? Apache version :- 2.0.59 Tomcat Version :- 5.5.9 Apache/Tomcat connectivity :- mod_jk Both of the Apache httpd and Tomcat versions that you mention above are very old (many years). You should upgrade to more recent versions. You do not indicate a version for mod_jk, but it is probably old also. when i have checked mod_jk.log file i am seeing this messages in the logs. [Mon Jan 10 08:14:52 2011] worker1 erib2bdev.ericsson.net 0.001122 [Mon Jan 10 08:14:55 2011] worker1 erib2bdev.ericsson.net 0.000732 [Mon Jan 10 08:14:57 2011] worker1 erib2bdev.ericsson.net 0.000740 [Mon Jan 10 08:15:00 2011] worker1 erib2bdev.ericsson.net 0.000846 [Mon Jan 10 08:15:02 2011] worker1 erib2bdev.ericsson.net 0.000952 These messages look to me as if, in your Apache httpd server, there are processes which communicate directly, through mod_jk, with a (real) external Internet server erib2bdev.ericsson.net (probably a Tomcat server also). That's all I can say. I do not know if that is your problem, or if these connections are a normal part of your application, or anything else. Only you can know that. Try to be a bit more specific about what the real problem is, and maybe we can start helping you. Come on, give us a clue. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Process going high even after restart of backend server tomcat
Hello Andre Warnier, Thanks for the response. My main problem is apache process is increasing very fast and reaching to count 259 thereby bringing donw application. For that i took thread dump also and tried to analyze but unable to find out what is causing the issue. I have also posted some part of thread dump in another post. I would liek to know why apache process is going so high in just 15 minuted and not coming down. I would need all experts help to resolve this issue. --- On Tue, 1/11/11, André Warnier a...@ice-sa.com wrote: From: André Warnier a...@ice-sa.com Subject: Re: Apache Process going high even after restart of backend server tomcat To: Tomcat Users List users@tomcat.apache.org Date: Tuesday, January 11, 2011, 5:28 PM Amol Puglia wrote: Hello Andre Warnier, Thanks for the response. Is there anyway i can bring down this connections. I would like to know why this process id's are not getting terminated automatically. I have run the traceroute command and following is the output of the command. Hi. After your other post, entitled Need help in understanding tomcat thread dump, it is a bit confusing now what your real problem is. That other post shows that you are runnig some java modules on your tomcat server, which seem to have something to do with the same supplier (Ericsson ?) than the one some Apache processes are connecting to using mod_jk. We have no idea here of your applications, what they are doing, if it is normal or not that they connect to an Internet server of the Ericsson company, and so on. Let's restart from the beginning, your original post : I am facing very weird problem of apache porcess going high upto 259 and increasing again even after restart of backend server ie. tomcat. So, is it the Apache httpd processes (children) that go very high (meaning that there are many, too many ?), or the back-end Tomcat threads ? What is the problem ? is it in Apache httpd, or in Tomcat ? does your server stop responding ? is it slow ? does it not do what it is supposed to do ? Apache version :- 2.0.59 Tomcat Version :- 5.5.9 Apache/Tomcat connectivity :- mod_jk Both of the Apache httpd and Tomcat versions that you mention above are very old (many years). You should upgrade to more recent versions. You do not indicate a version for mod_jk, but it is probably old also. when i have checked mod_jk.log file i am seeing this messages in the logs. [Mon Jan 10 08:14:52 2011] worker1 erib2bdev.ericsson.net 0.001122 [Mon Jan 10 08:14:55 2011] worker1 erib2bdev.ericsson.net 0.000732 [Mon Jan 10 08:14:57 2011] worker1 erib2bdev.ericsson.net 0.000740 [Mon Jan 10 08:15:00 2011] worker1 erib2bdev.ericsson.net 0.000846 [Mon Jan 10 08:15:02 2011] worker1 erib2bdev.ericsson.net 0.000952 These messages look to me as if, in your Apache httpd server, there are processes which communicate directly, through mod_jk, with a (real) external Internet server erib2bdev.ericsson.net (probably a Tomcat server also). That's all I can say. I do not know if that is your problem, or if these connections are a normal part of your application, or anything else. Only you can know that. Try to be a bit more specific about what the real problem is, and maybe we can start helping you. Come on, give us a clue. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Process going high even after restart of backend server tomcat
Amol Puglia wrote: Hello Andre Warnier, Thanks for the response. My main problem is apache process is increasing very fast and reaching to count 259 thereby bringing donw application. For that i took thread dump also and tried to analyze but unable to find out what is causing the issue. I have also posted some part of thread dump in another post. I would liek to know why apache process is going so high in just 15 minuted and not coming down. Amol, in this context the name Apache is confusing. There are 2 different packages, or servers : Apache httpd (a webserver, which can function as a front-end, using the mod_jk add-on module), and Apache Tomcat, which is a Java servlet engine (and also, confusingly, a webserver). Both can have threads. So what we would like to know first, is : which server are you talking about when you say apache process is increasing very fast and reaching to count 259 ? What is reaching count 259 and how do you see that ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Process going high even after restart of backend server tomcat
Hello Andre Warnier, Thanks for the response. I am talking about apache web server whose process count is increasiing to 259 and not coming down. I would like to know why process count is increasing and not coming down. Please help me to toublleshoot this issue.Let em know in case you need further information. --- On Tue, 1/11/11, André Warnier a...@ice-sa.com wrote: From: André Warnier a...@ice-sa.com Subject: Re: Apache Process going high even after restart of backend server tomcat To: Tomcat Users List users@tomcat.apache.org Date: Tuesday, January 11, 2011, 6:21 PM Amol Puglia wrote: Hello Andre Warnier, Thanks for the response. My main problem is apache process is increasing very fast and reaching to count 259 thereby bringing donw application. For that i took thread dump also and tried to analyze but unable to find out what is causing the issue. I have also posted some part of thread dump in another post. I would liek to know why apache process is going so high in just 15 minuted and not coming down. Amol, in this context the name Apache is confusing. There are 2 different packages, or servers : Apache httpd (a webserver, which can function as a front-end, using the mod_jk add-on module), and Apache Tomcat, which is a Java servlet engine (and also, confusingly, a webserver). Both can have threads. So what we would like to know first, is : which server are you talking about when you say apache process is increasing very fast and reaching to count 259 ? What is reaching count 259 and how do you see that ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Process going high even after restart of backend server tomcat
On 11/01/2011 13:06, Amol Puglia wrote: Hello Andre Warnier, Thanks for the response. I am talking about apache web server whose process count is increasiing to 259 and not coming down. I would like to know why process count is increasing and not coming down. Please help me to toublleshoot this issue.Let em know in case you need further information. Then you are on the wrong list. You should be using the httpd users mailing list. Mark --- On Tue, 1/11/11, André Warnier a...@ice-sa.com wrote: From: André Warnier a...@ice-sa.com Subject: Re: Apache Process going high even after restart of backend server tomcat To: Tomcat Users List users@tomcat.apache.org Date: Tuesday, January 11, 2011, 6:21 PM Amol Puglia wrote: Hello Andre Warnier, Thanks for the response. My main problem is apache process is increasing very fast and reaching to count 259 thereby bringing donw application. For that i took thread dump also and tried to analyze but unable to find out what is causing the issue. I have also posted some part of thread dump in another post. I would liek to know why apache process is going so high in just 15 minuted and not coming down. Amol, in this context the name Apache is confusing. There are 2 different packages, or servers : Apache httpd (a webserver, which can function as a front-end, using the mod_jk add-on module), and Apache Tomcat, which is a Java servlet engine (and also, confusingly, a webserver). Both can have threads. So what we would like to know first, is : which server are you talking about when you say apache process is increasing very fast and reaching to count 259 ? What is reaching count 259 and how do you see that ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Process going high even after restart of backend server tomcat
Amol Puglia wrote: Hello Andre Warnier, Thanks for the response. I am talking about apache web server whose process count is increasiing to 259 and not coming down. I would like to know why process count is increasing and not coming down. Please help me to toublleshoot this issue.Let em know in case you need further information. I am extremely tempted to answer that if the problem is at the Apache httpd server level, then it is not our problem on this list. We deal only with Tomcat. But someone else already answered that apparently. So I'll make another effort. To answer your question : we don't know. There can be a million reasons. Some of these reasons can have to do with the back-end Tomcat, but that is only a possibility. It may have nothing to do with Tomcat at all. One possibility is that there are just many user requests coming in to your Apache httpd server, and it is not fast enough to handle the load. So, in order to stay on this list, what makes you think that the problem has anything to do with the back-end Tomcat ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Process going high even after restart of backend server tomcat
In 'the other thread' there is still the question open of why there is Thread.sleep() in the MuxServlet. Can be a reason for a lot of open connections in the httpd server. But until there is more information about the problem I don't have anything to add to this discussion anymore. Ronald. Op dinsdag, 11 januari 2011 14:23 schreef André Warnier a...@ice-sa.com: Amol Puglia wrote: Hello Andre Warnier, Thanks for the response. I am talking about apache web server whose process count is increasiing to 259 and not coming down. I would like to know why process count is increasing and not coming down. Please help me to toublleshoot this issue.Let em know in case you need further information. I am extremely tempted to answer that if the problem is at the Apache httpd server level, then it is not our problem on this list. We deal only with Tomcat. But someone else already answered that apparently. So I'll make another effort. To answer your question : we don't know. There can be a million reasons. Some of these reasons can have to do with the back-end Tomcat, but that is only a possibility. It may have nothing to do with Tomcat at all. One possibility is that there are just many user requests coming in to your Apache httpd server, and it is not fast enough to handle the load. So, in order to stay on this list, what makes you think that the problem has anything to do with the back-end Tomcat ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Process going high even after restart of backend server tomcat
Hello Andre, When i posted same message to apache users list they said the apache process is going high because there is a problem in backend server tomcat and asked me to collect thread dump and post same in tomcat users list. so i posetd thread dump sanpshot in tomcat users list. It seems tomcat is not releasing connection and this is the reason apache proces count is goind high. I would like to know why tomcat is not releasing connection and wanted to figure out whether ther is problem in tomcat or application deployed on tomcat. I have taken therad dump and trying to analyze it but unable to understand what is causing problem by lloking at threads. Let em know if i can attache thread dump here so that experts from the forums can help me to understand thread dump. --- On Tue, 1/11/11, André Warnier a...@ice-sa.com wrote: From: André Warnier a...@ice-sa.com Subject: Re: Apache Process going high even after restart of backend server tomcat To: Tomcat Users List users@tomcat.apache.org Date: Tuesday, January 11, 2011, 6:53 PM Amol Puglia wrote: Hello Andre Warnier, Thanks for the response. I am talking about apache web server whose process count is increasiing to 259 and not coming down. I would like to know why process count is increasing and not coming down. Please help me to toublleshoot this issue.Let em know in case you need further information. I am extremely tempted to answer that if the problem is at the Apache httpd server level, then it is not our problem on this list. We deal only with Tomcat. But someone else already answered that apparently. So I'll make another effort. To answer your question : we don't know. There can be a million reasons. Some of these reasons can have to do with the back-end Tomcat, but that is only a possibility. It may have nothing to do with Tomcat at all. One possibility is that there are just many user requests coming in to your Apache httpd server, and it is not fast enough to handle the load. So, in order to stay on this list, what makes you think that the problem has anything to do with the back-end Tomcat ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Process going high even after restart of backend server tomcat
Amol Puglia wrote: Hello Andre, When i posted same message to apache users list they said the apache process is going high because there is a problem in backend server tomcat and asked me to collect thread dump and post same in tomcat users list. so i posetd thread dump sanpshot in tomcat users list. It seems tomcat is not releasing connection and this is the reason apache proces count is goind high. I would like to know why tomcat is not releasing connection and wanted to figure out whether ther is problem in tomcat or application deployed on tomcat. I have taken therad dump and trying to analyze it but unable to understand what is causing problem by lloking at threads. Let em know if i can attache thread dump here so that experts from the forums can help me to understand thread dump. Allright, so now we may start getting an idea of the problem. Maybe the first things to do : 1) do not post files as attachments in this list. Mostly, the server strips them. Instead, if we ask you for some text information, do a copy-and-paste, and paste the content directly in your email to the list. Also, when you copy contents of files here, please remove the comments, and any private information such as internet hostnames or passwords. 2) tell us what the platform is (O.S.) 3) if you can, also tell what exact version of the mod_jk moduile you are using. You should be able to find this in the message that Apache httpd prints in its error log when it starts. 3) in the configuration of Apache httpd, there must be configuration lines starting with Jk... Copy them here. 4) copy here the full content of the workers.properties file (comments removed) 5) in the Tomcat conf/server.xml file, there is a Connector element with a protocol=AJP. Copy this Connector element here. 6) if the platform is Unix/Linux, execute the following command and copy the result here : apache2ctl -V 7) if the platform is Unix/Linux, go to the bin directory of Tomcat, execute the command version.sh and copy the result here. With the above, we will get a much better idea of your configuration, and of what to ask next. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Process going high even after restart of backend server tomcat
On 10.01.2011 17:00, Amol Puglia wrote: Hello Team, I am facing very weird problem of apache porcess going high upto 259 and increasing again even after restart of backend server ie. tomcat. Apache version :- 2.0.59 Tomcat Version :- 5.5.9 Apache/Tomcat connectivity :- mod_jk when i have checked mod_jk.log file i am seeing this messages in the logs. [Mon Jan 10 08:14:52 2011] worker1 erib2bdev.ericsson.net 0.001122 [Mon Jan 10 08:14:55 2011] worker1 erib2bdev.ericsson.net 0.000732 [Mon Jan 10 08:14:57 2011] worker1 erib2bdev.ericsson.net 0.000740 [Mon Jan 10 08:15:00 2011] worker1 erib2bdev.ericsson.net 0.000846 [Mon Jan 10 08:15:02 2011] worker1 erib2bdev.ericsson.net 0.000952 As I already wrote on the httpd users list: The lines come from the JkRequestLogFormat configuration setting that you must have added somewhere (default: no such setting, no lines). The configured format will tell you, what the entry means. Likely it is either the name of the virtual host, or the value of the host header the client sent. Cleartext: this is requests being processed by your Apache server. The client sends these requests to erib2bdev.ericsson.net and they are received by your server. Or you configured your server to carry the name erib2bdev.ericsson.net. For details see: http://tomcat.apache.org/connectors-doc/reference/apache.html and look for JkRequestLogFormat in that page. The use of that directive is no longer recommended, because it is easier to include request log information into the usual access log via mod_log_config. Look for mod_log_config in the same page. I am not sure what does it mean when the name erib2bdev.ericsson.net is not refereed anywhere in my apache and tomcat configuration files and not even in workers.properties file. Can anyone help me torubleshoot this issue? Furthermore this might not have to do with your issue, namely that you think the number of Apache processes is to big. From the other thread you posted containing the Java thread Dump excerpt we think that the reason for the growing number of processes is, that your MUXServlet doesn't handle the request and instead enters sleep. To make this assumption a fact, you would need to tell us, whether there are many threads in the same dump, that have the same stack (sleep in MUXServlet), more precisely how many and what the names of the threads are. If you find let's say 200 of those, then you are close to an explanation. Finally you have to do more than one dump to understand, whether the sleep() was only lasting for a short time, or it is happening all the time. Then you have to start talking to the developers of the MUXServlet showing them those dumps. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Apache Process going high even after restart of backend server tomcat
Hello Team, I am facing very weird problem of apache porcess going high upto 259 and increasing again even after restart of backend server ie. tomcat. Apache version :- 2.0.59 Tomcat Version :- 5.5.9 Apache/Tomcat connectivity :- mod_jk when i have checked mod_jk.log file i am seeing this messages in the logs. [Mon Jan 10 08:14:52 2011] worker1 erib2bdev.ericsson.net 0.001122 [Mon Jan 10 08:14:55 2011] worker1 erib2bdev.ericsson.net 0.000732 [Mon Jan 10 08:14:57 2011] worker1 erib2bdev.ericsson.net 0.000740 [Mon Jan 10 08:15:00 2011] worker1 erib2bdev.ericsson.net 0.000846 [Mon Jan 10 08:15:02 2011] worker1 erib2bdev.ericsson.net 0.000952 I am not sure what does it mean when the name erib2bdev.ericsson.net is not refereed anywhere in my apache and tomcat configuration files and not even in workers.properties file. Can anyone help me torubleshoot this issue? Thanks in advance.
Re: Apache Process going high even after restart of backend server tomcat
Amol Puglia wrote: Hello Team, I am facing very weird problem of apache porcess going high upto 259 and increasing again even after restart of backend server ie. tomcat. Apache version :- 2.0.59 Tomcat Version :- 5.5.9 Apache/Tomcat connectivity :- mod_jk when i have checked mod_jk.log file i am seeing this messages in the logs. [Mon Jan 10 08:14:52 2011] worker1 erib2bdev.ericsson.net 0.001122 [Mon Jan 10 08:14:55 2011] worker1 erib2bdev.ericsson.net 0.000732 [Mon Jan 10 08:14:57 2011] worker1 erib2bdev.ericsson.net 0.000740 [Mon Jan 10 08:15:00 2011] worker1 erib2bdev.ericsson.net 0.000846 [Mon Jan 10 08:15:02 2011] worker1 erib2bdev.ericsson.net 0.000952 I am not sure what does it mean when the name erib2bdev.ericsson.net is not refereed anywhere in my apache and tomcat configuration files and not even in workers.properties file. Can anyone help me torubleshoot this issue? What I can tell you, is that the above is a real system on the Internet. I suggest trying : nslookup erib2bdev.ericsson.net traceroute erib2bdev.ericsson.net Then try http://www.whois.net/whois/ericsson.net and then try to figure out why an application on your Apache server has its own mod_jk configuration, and calls back to this Internet server every 2 seconds or so.. Such as, have you paid your phone bill ? ;-) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
