Re: Tomcat Thread utilization growing very large causing a hung tomcat instance
Hi Peter, Thanks for your resonse. I believe the issue is why is the Tomcat instance utilizing around 400+ threads. It should be just utilizing only 200 threads because of the default value for maxThreads? On the other servers, Thread utilization is at 200-250 on normal operations which is still over 200. The system was designed by the solution architech to use 1.5GB for heap and we have observed the memory usage to go as high as 1.2GB per tomcat instance. The proposal is to shutdown one of the Tomcat instances to free up memory but I think the root issue is why we are reaching up to 400 threads. We are planning another round of load tests but I am not familiar with how to analyze thread dumps. Any ideas or documentation I can look into? Also apologies if you got my first email multiple times. Might have been a gmail issue as I only sent out one email to the list but was notified by one of the people in the list of the multiple messsages. Best regards, On Tue, Aug 3, 2010 at 11:04 PM, Peter Crowther wrote: > What happens if you *reduce* the allocated heap size - do you actually need > that much Java heap? Native threads are created outside of the Java heap. > You have 2G for (4 * 400) = 1600 threads plus the OS, or about 1 Mbyte per > thread (this assumes no swap space). That may not be enough. > > - Peter > > On 3 August 2010 15:21, Tomcat Novice wrote: > > > Hi, > > > > I hope someone can help me. I am new to tomcat and I have been > encountering > > issues where a particular tomcat instance/s are having large thread count > > during load testing and results in a hung state. The error shown on the > > browser is: java.lang.OutOfMemoryError: unable to create new native > thread > > (stack trace attached) > > > > Environment: > > > > OS: Windows 2003 64 bit > > JDK 1.5 > > 8GB Total Machine RAM > > 4 Tomcat 5.5 instances running as a service on one Physical Server (each > > Tomcat configured to use 1.5GB of Heap) 4 instances x 1.5GB = 6GB of Heap > > 4 Identical servers configured the same way and load balanced > > IIS Web Server front end (AJP Port configured) no maxThreads property is > > set on server.xml for AJP port declaration. Per documentation ( > > http://tomcat.apache.org/tomcat-5.5-doc/config/ajp.html), if no value is > > set maxThreads are set to 200 > > > > Not all 4 servers are experiencing this problem. > > > > 1. Any ideas how I can resolve/isolate this problem? > > 2. I know I need to do a thread dump but I am not familiar on how to > > analyze thread dumps from Tomcat. Can you point me to documententation on > > how I can analyze thread dumps from Tomcat? > > 3. Also, can someone confirm if the thread dumps from JDK 1.5 are the > same > > as the Thread Dumps from a JDK 1.6? I seem to be getting different > formats. > > Can someone just confirm? > > 4. If the maxThread is set to 200, why are the Tomcat instances utilizing > > up to 400 Threads (see screenshot) > > > > Regards, > > > > Tomcat Novice > > > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > >
RE: Tomcat Thread utilization growing very large causing a hung tomcat instance
> From: Tomcat Novice [mailto:tomcat.nov...@gmail.com] > Subject: Tomcat Thread utilization growing very large causing a hung > tomcat instance > > I have been encountering issues where a particular tomcat > instance/s are having large thread count during load testing > and results in a hung state. I suspect that your code is spawning threads, and not managing them properly. Your horribly formatted stack trace shows timer threads being started by: at com.hp.ov.sm.client.common.communications.HeartBeat.(HeartBeat.java:68) at com.hp.ov.sm.client.common.communications.SOAPClient.connect(SOAPClient.java:237) at com.hp.ov.sm.client.webtier.WebClient.connect(WebClient.java:95) at com.hp.ov.sm.client.webtier.WebController.init(WebController.java:599) at com.hp.ov.sm.client.webtier.WebControllerServlet.index(WebControllerServlet.java:448) at com.hp.ov.sm.client.webtier.WebControllerServlet.process(WebControllerServlet.java:366) at com.hp.ov.sm.client.webtier.WebControllerServlet.processHTTPRequest(WebControllerServlet.java:317) at com.hp.ov.sm.client.webtier.WebControllerServlet.doGet(WebControllerServlet.java:255) (The next time, leave the line breaks in the trace, please.) > 1. Any ideas how I can resolve/isolate this problem? Fix your webapp to manage threads properly. > 2. I know I need to do a thread dump but I am not familiar on how to > analyze thread dumps from Tomcat. Can you point me to documententation > on how I can analyze thread dumps from Tomcat? Take a thread dump on a Tomcat that's been used but now idle, and then one on the problematic system, and compare. > 3. Also, can someone confirm if the thread dumps from JDK 1.5 are the > same as the Thread Dumps from a JDK 1.6? I seem to be getting different > formats. Can someone just confirm? I don't recall any major differences, but they are unlikely to be identical, since minor enhancements to the JVM are made all the time. > 4. If the maxThread is set to 200, why are the Tomcat instances > utilizing up to 400 Threads (see screenshot) There isn't a screenshot (the mailing list strips most attachments). Post data, not pictures. Also post your server.xml so we can see what you have configured, but please remove the commented-out bits (and passwords) before doing so. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Thread utilization growing very large causing a hung tomcat instance
What happens if you *reduce* the allocated heap size - do you actually need that much Java heap? Native threads are created outside of the Java heap. You have 2G for (4 * 400) = 1600 threads plus the OS, or about 1 Mbyte per thread (this assumes no swap space). That may not be enough. - Peter On 3 August 2010 15:21, Tomcat Novice wrote: > Hi, > > I hope someone can help me. I am new to tomcat and I have been encountering > issues where a particular tomcat instance/s are having large thread count > during load testing and results in a hung state. The error shown on the > browser is: java.lang.OutOfMemoryError: unable to create new native thread > (stack trace attached) > > Environment: > > OS: Windows 2003 64 bit > JDK 1.5 > 8GB Total Machine RAM > 4 Tomcat 5.5 instances running as a service on one Physical Server (each > Tomcat configured to use 1.5GB of Heap) 4 instances x 1.5GB = 6GB of Heap > 4 Identical servers configured the same way and load balanced > IIS Web Server front end (AJP Port configured) no maxThreads property is > set on server.xml for AJP port declaration. Per documentation ( > http://tomcat.apache.org/tomcat-5.5-doc/config/ajp.html), if no value is > set maxThreads are set to 200 > > Not all 4 servers are experiencing this problem. > > 1. Any ideas how I can resolve/isolate this problem? > 2. I know I need to do a thread dump but I am not familiar on how to > analyze thread dumps from Tomcat. Can you point me to documententation on > how I can analyze thread dumps from Tomcat? > 3. Also, can someone confirm if the thread dumps from JDK 1.5 are the same > as the Thread Dumps from a JDK 1.6? I seem to be getting different formats. > Can someone just confirm? > 4. If the maxThread is set to 200, why are the Tomcat instances utilizing > up to 400 Threads (see screenshot) > > Regards, > > Tomcat Novice > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org >
Tomcat Thread utilization growing very large causing a hung tomcat instance
Hi, I hope someone can help me. I am new to tomcat and I have been encountering issues where a particular tomcat instance/s are having large thread count during load testing and results in a hung state. The error shown on the browser is: java.lang.OutOfMemoryError: unable to create new native thread (stack trace attached) Environment: OS: Windows 2003 64 bit JDK 1.5 8GB Total Machine RAM 4 Tomcat 5.5 instances running as a service on one Physical Server (each Tomcat configured to use 1.5GB of Heap) 4 instances x 1.5GB = 6GB of Heap 4 Identical servers configured the same way and load balanced IIS Web Server front end (AJP Port configured) no maxThreads property is set on server.xml for AJP port declaration. Per documentation ( http://tomcat.apache.org/tomcat-5.5-doc/config/ajp.html), if no value is set maxThreads are set to 200 Not all 4 servers are experiencing this problem. 1. Any ideas how I can resolve/isolate this problem? 2. I know I need to do a thread dump but I am not familiar on how to analyze thread dumps from Tomcat. Can you point me to documententation on how I can analyze thread dumps from Tomcat? 3. Also, can someone confirm if the thread dumps from JDK 1.5 are the same as the Thread Dumps from a JDK 1.6? I seem to be getting different formats. Can someone just confirm? 4. If the maxThread is set to 200, why are the Tomcat instances utilizing up to 400 Threads (see screenshot) Regards, Tomcat Novice java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:574) at java.util.Timer.(Timer.java:137) at java.util.Timer.(Timer.java:106) at com.hp.ov.sm.client.common.communications.HeartBeat.(HeartBeat.java:68) at com.hp.ov.sm.client.common.communications.SOAPClient.connect(SOAPClient.java:237) at com.hp.ov.sm.client.webtier.WebClient.connect(WebClient.java:95) at com.hp.ov.sm.client.webtier.WebController.init(WebController.java:599) at com.hp.ov.sm.client.webtier.WebControllerServlet.index(WebControllerServlet.java:448) at com.hp.ov.sm.client.webtier.WebControllerServlet.process(WebControllerServlet.java:366) at com.hp.ov.sm.client.webtier.WebControllerServlet.processHTTPRequest(WebControllerServlet.java:317) at com.hp.ov.sm.client.webtier.WebControllerServlet.doGet(WebControllerServlet.java:255) at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.compressionFilters.CompressionFilter.doFilter(CompressionFilter.java:203) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:286) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149) at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265) at com.hp.ov.cwc.web.CacheControlFilter.doFilter(CacheControlFilter.java:70) at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275) at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149) at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter
