On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul <subhro.p...@tcs.com> wrote:
> Dear Team, > > One of our client's website stopped working yesterday. We observed that > Tomcat servers were not working properly during that time. We have checked > the memory usage of the server was fine but in the Catalina.out log we > found it was already reached to max thread which is 512 though the number > of connections to the server was normal. We took a thread dump from the > server using VisualVM and we got the below message from threaddump: > Since a thread dump is a point in time snapshot, you should always take multiple thread dumps, with a few seconds in between each one. This gives you additional perspective as to what's happening with the threads over a period of time. > > "http-8080-1" - Thread t@22 > > java.lang.Thread.State: BLOCKED > > at java.util.Vector$1.nextElement(Vector.java:320) > > - waiting to lock <37749687> (a java.util.Vector) owned by > "http-8080-116" t@161 > > at > org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116) > > at > org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314) > Look at what header.jsp is doing. It seems to be doing something with the Vector class which is causing the thread to block. > > at > org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > at > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) > > at > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) > > at > org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > > at > org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) > > at > org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551) > > at > org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488) > > at > org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968) > > at > org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126) > > at > org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > at > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) > > at > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) > > at > org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > > at > org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269) > > at > org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.java:81) > > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) > > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) > > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > > at java.lang.Thread.run(Thread.java:701) > > > > Locked ownable synchronizers: > > - None > > > > This was coming for different threads. Once we restarted the servers, the > website back to normal again but we got the below exception in the log : > > > > Apr 22, 2015 11:15:28 AM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads > > SEVERE: A web application appears to have started a thread named > [http-8080-1] but has failed to stop it. This is very likely to create a > memory leak. > > This means your app started a thread and failed to properly stop it. If you stopped the process completely, this is not going to cause any problems because the entire process would have exited and so this thread would go away any way. If you're doing hot redeploy's (i.e. deplying apps without restarting the process), this could cause problems. Either way, when you have a moment you should try to investigate what is creating that thread and setup something to properly close it. > > > So, we want to know while the thread is blocked is it like deadlock > condition for which all threads were unavailable? The thread dump simply lists the thread as "blocked". This generally means that at some point it will complete, so it's not technically a deadlock. If you had multiple thread dumps, you could watch the thread across all of them and see how it progresses. > Current thread count I about 190 but after few days this will reach to > 500+ again even if the concurrent users are not high. Memory usage of the > server was normal during this issue. This problem is happening from last 2 > 3 months. > Look at the header.jsp, especially anything that has changed in the last 2 - 3 months. Dan > > > > Thanks & Regards, > > Subhro Paul > > =====-----=====-----===== > Notice: The information contained in this e-mail > message and/or attachments to it may contain > confidential or privileged information. If you are > not the intended recipient, any dissemination, use, > review, distribution, printing or copying of the > information contained in this e-mail message > and/or attachments to it are strictly prohibited. If > you have received this communication in error, > please notify us by reply e-mail or telephone and > immediately and permanently delete the message > and any attachments. Thank you > > >
