RE: Memory leak in Tomcat 5.5.16
Hello Tom, I am facing a similar issue. Could you please send that piece of code that was causing you this error, just that snippet should be good. Thanks Tom Price-3 wrote: Hi all, No more help required - I traced back all the references to the Request objects and it did turn out to be a bug in my application code. One of my tracing classes (written a long time ago, before we used Tomcat) was caching all created Thread objects. So when Tomcat decided to allocate new threads to process incoming requests (and un-referenced the old ones), my application cached the old threads and they slowly leaked. I'm assuming that Tomcat only does this when handling multiple concurrent connections (i.e. threads are reused when handling single-threaded connections), hence the fact that I could only reproduce the problem when sending in concurrent requests. Thanks for everyone's suggestions. Tom. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- View this message in context: http://old.nabble.com/Memory-leak-in-Tomcat-5.5.16-tp16335734p28454415.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Memory leak in Tomcat 5.5.16
Hi all, No more help required - I traced back all the references to the Request objects and it did turn out to be a bug in my application code. One of my tracing classes (written a long time ago, before we used Tomcat) was caching all created Thread objects. So when Tomcat decided to allocate new threads to process incoming requests (and un-referenced the old ones), my application cached the old threads and they slowly leaked. I'm assuming that Tomcat only does this when handling multiple concurrent connections (i.e. threads are reused when handling single-threaded connections), hence the fact that I could only reproduce the problem when sending in concurrent requests. Thanks for everyone's suggestions. Tom. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Memory leak in Tomcat 5.5.16
Hi, I have now managed to do some analysis of the classes that are referencing the leaked objects, can anyone help me interpret these results? I got jmap/jhat working by upgrading to JRE 1.6.0_05, and took a heap dump at a time after a period of stress when all SOAP/XML connections had been closed (so no requests were outstanding). I had also forced full garbage collection a few times. There were many org.apache.tomcat.util.buf.MessageBytes objects still referenced, which all seem to be held directly and indirectly by org.apache.coyote.Request objects. These are referenced by several classes, which is where I get a bit lost: [EMAIL PROTECTED] (70 bytes) : field request [EMAIL PROTECTED] (120 bytes) : field request [EMAIL PROTECTED] (153 bytes) : field coyoteRequest [EMAIL PROTECTED] (68 bytes) : field req [EMAIL PROTECTED] (78 bytes) : field coyoteRequest [EMAIL PROTECTED] (78 bytes) : field req Where do I go next to find out who is keeping these Request objects alive? Thanks, Tom. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory leak in Tomcat 5.5.16
Tom Price I'm trying to find a tool that will show me which objects are holding onto references to these Tomcat objects, but haven't had much success so far - any suggestions gratefully received. Not free, but I use YourKit and get on very well with it. Mark - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Memory leak in Tomcat 5.5.16
I use Netbeans 6 with its built-in profiler (all free) to track down memory usage issues, and other performance problems in Java apps in general and Tomcat in particular. Hai Vu Tom Price [EMAIL PROTECTED] 27/03/2008 07:59 PM Please respond to Tomcat Users List users@tomcat.apache.org To Tomcat Users List users@tomcat.apache.org cc Subject RE: Memory leak in Tomcat 5.5.16 Christopher, Christopher Schultz wrote: Have you been able to compare the numbers of those objects after, say, 100 requests with the same object counts after, say, 1 requests? It /is/ possible that Tomcat is leaking memory per connection, but very unlikely given that thousands of servers are humming along quite nicely without such problems. Thanks for the suggestions. I have looked at the change in number of objects over time, and the only objects that appear to be leaked consistently are the following (the number is the increase in object count over a typical 10 minute period, with about 2000 requests): org.apache.tomcat.util.buf.ByteChunk 10437 org.apache.tomcat.util.buf.CharChunk 8946 org.apache.tomcat.util.buf.MessageBytes 7881 Other object types, including byte[] and char[], vary wildly in each 10 minute period, presumably depending on exactly what's going on at the time of the sample. I'm trying to find a tool that will show me which objects are holding onto references to these Tomcat objects, but haven't had much success so far - any suggestions gratefully received. The only really good way to figure out what's happening is to instrument your JVM and then test the hell out of it. Use JMeter to generate load (serial or concurrent) and observe the heap, etc. You might even try to disable some of the SOAP methods you are testing to see if this leak only occurs with certain calls, or combinations of calls. That's what I'm doing now... I will also try Filip's suggestion of disabling string caching. Sadly jmap doesn't work on this system because the heap is large and jmap always crashes with an OutOfMemoryError due to the 2GB heap limit in the JVM. I haven't used JMeter before but will look into it. Thanks, Tom. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory leak in Tomcat 5.5.16
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tom, Tom Price wrote: | I have analyzed the heap usage on a | system where this memory leak has occurred, and I see that the following | classes take up most of the memory: [snip] | Does anyone know what could trigger the above classes to be leaked in | Tomcat 5.5.16? I have also tried upgrading to Tomcat 5.5.26, but the | problem still occurs. Not to dismiss your problem, but these kinds of things tend to be problems with the application rather than problems with Tomcat. Are you able to use a profiler that can tell you where the object allocations are actually occurring? It's not unusual to have a ton of byte[] objects, char[] objects, String objects, and Class objects. I'm guessing that the MessageBytes, ByteChunk, and CharChunk objects in your heap summary are all related to the byte[], char[], and String objects you can see there as well. Have you been able to compare the numbers of those objects after, say, 100 requests with the same object counts after, say, 1 requests? It /is/ possible that Tomcat is leaking memory per connection, but very unlikely given that thousands of servers are humming along quite nicely without such problems. It's also possible that Axis itself is doing some type of caching or actually leaking memory somehow, but I also doubt that it does. The only really good way to figure out what's happening is to instrument your JVM and then test the hell out of it. Use JMeter to generate load (serial or concurrent) and observe the heap, etc. You might even try to disable some of the SOAP methods you are testing to see if this leak only occurs with certain calls, or combinations of calls. Good luck, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkfr32wACgkQ9CaO5/Lv0PB5hACfY4Q88EAqc6ecvgOXKPwS3xtA ufMAn3tSIziYrq8sNcw2CzTnw08rOoqF =EgBf -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory leak in Tomcat 5.5.16
try to disable the cache in conf/catalina.properties, at the bottom there is a setting you can set to false tomcat.util.buf.StringCache.byte.enabled=true if the problem still exists, use jmap to dump the heap, and analyse the actual dependencies for mem usage Filip Tom Price wrote: Hi, I have a Java application that exposes a web services (SOAP/XML) interface using Apache Tomcat and Axis2 (see detailed background below). When this interface is heavily utilized, the application gradually leaks old space memory until it runs out. I have analyzed the heap usage on a system where this memory leak has occurred, and I see that the following classes take up most of the memory: SizeCount Class description --- 433059616 557692 char[] 292410472 124245 byte[] 1780156087232 * ConstMethodKlass 16543576295421 org.apache.tomcat.util.buf.MessageBytes 15600840390021 org.apache.tomcat.util.buf.ByteChunk 13372920334323 org.apache.tomcat.util.buf.CharChunk 11575656482319 java.lang.String Does anyone know what could trigger the above classes to be leaked in Tomcat 5.5.16? I have also tried upgrading to Tomcat 5.5.26, but the problem still occurs. This is very similar to a memory leak posted to this list in 2006, but I can't find what the resolution was in that case: http://readlist.com/lists/tomcat.apache.org/users/5/27073.html I have also consulted http://wiki.apache.org/tomcat/OutOfMemory but couldn't find the solution. The problem only seems to occur when multiple concurrent requests are being received - if I issue requests in a single threaded manner, the server can handle many hundreds of thousands of requests over an extended period without any apparent memory leakage. But if multiple requests are sent concurrently, the application leaks about 1MB per 100 requests. Here are the details of the application: OS: Solaris 10 (Intel) JRE: 1.5.0_07, launched from native C code (not the Java executable) Tomcat: 5.5.16, embedded within my Java application Axis2: 1.3 Thanks in advance for any help. Tom. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Memory leak in Tomcat 5.5.16
Christopher, Christopher Schultz wrote: Have you been able to compare the numbers of those objects after, say, 100 requests with the same object counts after, say, 1 requests? It /is/ possible that Tomcat is leaking memory per connection, but very unlikely given that thousands of servers are humming along quite nicely without such problems. Thanks for the suggestions. I have looked at the change in number of objects over time, and the only objects that appear to be leaked consistently are the following (the number is the increase in object count over a typical 10 minute period, with about 2000 requests): org.apache.tomcat.util.buf.ByteChunk10437 org.apache.tomcat.util.buf.CharChunk8946 org.apache.tomcat.util.buf.MessageBytes 7881 Other object types, including byte[] and char[], vary wildly in each 10 minute period, presumably depending on exactly what's going on at the time of the sample. I'm trying to find a tool that will show me which objects are holding onto references to these Tomcat objects, but haven't had much success so far - any suggestions gratefully received. The only really good way to figure out what's happening is to instrument your JVM and then test the hell out of it. Use JMeter to generate load (serial or concurrent) and observe the heap, etc. You might even try to disable some of the SOAP methods you are testing to see if this leak only occurs with certain calls, or combinations of calls. That's what I'm doing now... I will also try Filip's suggestion of disabling string caching. Sadly jmap doesn't work on this system because the heap is large and jmap always crashes with an OutOfMemoryError due to the 2GB heap limit in the JVM. I haven't used JMeter before but will look into it. Thanks, Tom. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
