-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Saumil,
On 4/1/13 11:01 PM, saumil shah wrote: > I recently deployed one of the COTS products SAP Business Objects. > When the product was deployed , everything seemed to run fine but > yesterday we started experiencing "Service is unavailable" error , > upon enabling DEBUG logs in Tomcat , we saw the error below with > memory leak message. We were wondering what could have caused it > .... > > The Tomcat is 64 bit , JVM is 64 bit but the applications deployed > are 32 bit webapps .... we say Tomcat6 process becoming > unresponsive around 2GB mark. > > > Apr 1, 2013 8:53:18 PM org.apache.coyote.http11.Http11Protocol > pauseINFO: Pausing Coyote HTTP/1.1 on http-8080Apr 1, 2013 8:53:19 > PM org.apache.catalina.core.StandardService stopINFO: Stopping > service CatalinaApr 1, 2013 8:53:19 PM > org.apache.catalina.loader.WebappClassLoader > clearReferencesThreadsSEVERE: The web application > [/AnalyticalReporting] appears to have started a thread named > [Business Objects - Sessions Clean up] but has failed to stop it. > This is very likely to create a memory leak. This means what it says it means: your webapp (not Tomcat) started a thread that wasn't stopped by the time the webapp was stopped. This will pin the webapp's ClassLoader in memory and you'll get a whole bunch of stuff that can't be GC'd. The simple explanation is that you need to be sure that your threads are stopped. The solution may be complicated. Presumably any component that starts a thread also knows how to stop that same thread. Check the documentation for those components to make sure that you are shutting them down properly (usually in a ServletContextListener, etc.). Sometimes, it takes a few seconds for a thread to complete its work and actually terminate, and the messages you get from Tomcat are inaccurate. That's for you to determine. > Apr 1, 2013 8:53:19 PM > org.apache.catalina.loader.WebappClassLoader > clearReferencesThreadsSEVERE: The web application > [/AnalyticalReporting] appears to have started a thread named > [AWT-Windows] but has failed to stop it. This is very likely to > create a memory leak. Read the documentation for the JreMemoryLeakLeakPreventionListener. This is a standard Tomcat component that can be used to work-around well-known memory leaks, etc. that are triggered by certain JVM calls. There are specific configurations for preventing AWT threads from being problematic, but they are not enabled by default. You probably want to enable them if you are using AWT for anything. > Apr 1, 2013 8:53:19 PM > org.apache.catalina.loader.WebappClassLoader > clearReferencesThreadsSEVERE: The web application > [/AnalyticalReporting] appears to have started a thread named > [ORBacus:Client:SenderThread] but has failed to stop it. This is > very likely to create a memory leak. Anything with "ORB" in the name usually means a CORBA-related component: check to see that anything that start threads is also stopping them. > Apr 1, 2013 8:53:19 PM > org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMapSEVERE: The web application > [/AnalyticalReporting] created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@63e3d2dd]) > and a value of type [com.businessobjects.wp.xml.jaxp.XMLJaxpParser] > (value [com.businessobjects.wp.xml.jaxp.XMLJaxpParser@228de67]) but > failed to remove it when the web application was stopped. This is > very likely to create a memory leak. Looks like your Business Objects version has a bug in it that does not clear-out ThreadLocal values when requests are completed. Talk to them about this problem. > Apr 1, 2013 8:53:20 PM > org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMapSEVERE: The web application [/VoyagerClient] > created a ThreadLocal with key of type > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl] > > (value > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl@13123b1c]) > > and a value of type > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl.Counter] > > (value > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl$Counter@11e6b50c]) > > but failed to remove it when the web application was stopped. This is > very likely to create a memory leak.Apr 1, 2013 8:53:20 PM > org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMapSEVERE: The web application [/VoyagerClient] > created a ThreadLocal with key of type > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl] > > (value > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl@41783002]) > > and a value of type > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl.Counter] > > (value > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl$Counter@3997f4e2]) > > but failed to remove it when the web application was stopped. This is > very likely to create a memory leak. Looks like AspectJ has a similar problem: ThreadLocals need to be removed from threads after every request. You are probably seeing a pattern, here, so I won't bother looking at every other error message you have posted. - -chris -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRWzh6AAoJEBzwKT+lPKRYUoQQALSx4IVs9Pd/QfAQKeB6klIP 9bfoG8bRk/fcqdQiJfaNJGw8xaKm0WVMalYB/U1xH7dZSdFDlz4LoQuitRLEMnsX WyRspvI3N3KUIhGUuU9GiVz0nn1vwaQF87K81g/7IpCOKcFsWfruHnict2UMjrg1 pEI7xiNoC3oPsbd6+APhAy9JUip8gt4j7I0VUVAvCOg4L7PLvKGY3y4TXy594uFO D/QZ6cN/9SdImOO9wunwHk1+Ot3B3ZKuurdbJk6djXyt6K2Et4lHGJ8KldhZfgfj QdQjRg/ybTune+gWJ9QTPo5p7FOgTJxS2x94a+pXx9aTkCBqbBiKCWWU47O+wtf4 P5Nc71m0N1zeLsSjThjDoaH8DA77AbwtNWHQEwemvqp6y8CQy8ypp/f58lrzdOza lFhER7mMtzlNNCHdxZKUEY9yBaEIzWAV9vPP51yWEUBCJaGRZ9icTZNvIQPUAcLB JzoIfgEHFjrVlLXCKb0phIq6He0B/yDZ4xwA2rn9bXKhRJEuZbBoYXTM6jsRJDNk ab/wMZKXfK5On5N9U9TVj3kzGni35GI2wApJ9OGvYWdKv+9oX4vKQNOl8xAdm8IM wAE3ooZk0XvsotLphMP+V9PHXNCaeZeWORsNKKUlkxZDAPU5vdmaVfcdU0um2lm2 yqAz4cJjThxGwbh62xe3 =UtMt -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
