Dear Team, One of our client's website stopped working yesterday. We observed that Tomcat servers were not working properly during that time. We have checked the memory usage of the server was fine but in the Catalina.out log we found it was already reached to max thread which is 512 though the number of connections to the server was normal. We took a thread dump from the server using VisualVM and we got the below message from threaddump:
"http-8080-1" - Thread t@22 java.lang.Thread.State: BLOCKED at java.util.Vector$1.nextElement(Vector.java:320) - waiting to lock <37749687> (a java.util.Vector) owned by "http-8080-116" t@161 at org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116) at org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968) at org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspService(my_005fbill_jsp.java:126) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269) at org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.java:81) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:701) Locked ownable synchronizers: - None This was coming for different threads. Once we restarted the servers, the website back to normal again but we got the below exception in the log : Apr 22, 2015 11:15:28 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: A web application appears to have started a thread named [http-8080-1] but has failed to stop it. This is very likely to create a memory leak. So, we want to know while the thread is blocked is it like deadlock condition for which all threads were unavailable? Current thread count I about 190 but after few days this will reach to 500+ again even if the concurrent users are not high. Memory usage of the server was normal during this issue. This problem is happening from last 2 3 months. Thanks & Regards, Subhro Paul =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
