Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 16/11/2010 05:08, Brian wrote: I still have to find the reason of the leak. Tomcat attempts to find notify you of potential memory leaks. You reported messages in your logs at app reload. Did you tell us what they are yet? p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Hu Pid, Now that I looked, I did find something! Nov 15, 2010 8:37:28 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Nov 15, 2010 8:37:28 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Nov 15, 2010 8:37:28 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Nov 15, 2010 8:37:29 PM org.apache.catalina.loader.WebappClassLoader loadClass INFO: Illegal access: this web application instance has been stopped already. Could not load com.manuals.common.utility.Log. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact. java.lang.IllegalStateException at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1531) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1491) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) at com.manuals.model.utility.DatabaseObject.getStatement(DatabaseObject.java:434) at com.manuals.model.utility.DatabaseObject.getQueryResultSet(DatabaseObject.java:366) at com.manuals.model.utility.DatabaseObject.getQueryResultSet(DatabaseObject.java:326) And also this: ov 15, 2010 8:31:09 PM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 1 instance(s) to be deallocated Nov 15, 2010 8:31:10 PM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 1 instance(s) to be deallocated Nov 15, 2010 8:31:11 PM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 1 instance(s) to be deallocated Nov 15, 2010 8:31:12 PM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 1 instance(s) to be deallocated Nov 15, 2010 8:31:13 PM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 1 instance(s) to be deallocated Nov 15, 2010 8:31:14 PM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 1 instance(s) to be deallocated Nov 15, 2010 8:31:16 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] appears to have started a thread named [MySQL Statement Cancellation Timer] but has failed to stop it. This is very likely to create a memory leak. Nov 15, 2010 8:31:20 PM org.apache.catalina.core.StandardContext stop INFO: Container org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/] has not been started Nov 15, 2010 8:31:21 PM org.apache.catalina.startup.HostConfig checkResources INFO: Undeploying context [] -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Tuesday, November 16, 2010 03:43 AM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 16/11/2010 05:08, Brian wrote: I still have to find the reason of the leak. Tomcat attempts to find notify you of potential memory leaks. You reported messages in your logs at app reload. Did you tell us what they are yet? p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Brian, On 11/16/2010 3:23 PM, Brian wrote: Nov 15, 2010 8:37:28 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Looks like you have some (or at least one) long-running request. Maybe it's generating a ton of buffered output, too :) Nov 15, 2010 8:37:28 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Again. Nov 15, 2010 8:37:28 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Interesting that it generated all of these messages at the same time. I'm not sure if that means you have 3 requests pending or that 3 different problems were found with one thread. Nov 15, 2010 8:37:29 PM org.apache.catalina.loader.WebappClassLoader loadClass INFO: Illegal access: this web application instance has been stopped already. Could not load com.manuals.common.utility.Log. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact. java.lang.IllegalStateException at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1531) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1491) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) at com.manuals.model.utility.DatabaseObject.getStatement(DatabaseObject.java:434) at com.manuals.model.utility.DatabaseObject.getQueryResultSet(DatabaseObject.java:366) at com.manuals.model.utility.DatabaseObject.getQueryResultSet(DatabaseObject.java:326) So that thread is actually doing something -- it hasn't just blocked indefinitely: your DatabaseObject class is causing the WebappClassLoader to attempt to load a new class that hadn't been previously loaded. Since WebappClassLoader knows it's shutting down (or has already shut down at that point), it is refusing to fulfill the request. And also this: ov 15, 2010 8:31:09 PM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 1 instance(s) to be deallocated Nov 15, 2010 8:31:10 PM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 1 instance(s) to be deallocated Nov 15, 2010 8:31:11 PM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 1 instance(s) to be deallocated Nov 15, 2010 8:31:12 PM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 1 instance(s) to be deallocated Nov 15, 2010 8:31:13 PM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 1 instance(s) to be deallocated Nov 15, 2010 8:31:14 PM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 1 instance(s) to be deallocated Since these messages are from earlier, I assume you were just pasting them in arbitrary order into your post: it looks like Tomcat waited for a while for your requests to complete (looping, complaining once per second) and then finally gave up and issued the messages above (...still processing a request...) at the end. Nov 15, 2010 8:31:16 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] appears to have started a thread named [MySQL Statement Cancellation Timer] but has failed to stop it. This is very likely to create a memory leak. There is almost nothing you can do about this except to upgrade your MySQL driver: http://bugs.mysql.com/bug.php?id=36565 Nov 15, 2010 8:31:20 PM org.apache.catalina.core.StandardContext stop INFO: Container org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/] has not been started This is a different issue. Note that [] (root) is not [/]. Have you specified your webapp's path somewhere. Don't do that. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzi+qwACgkQ9CaO5/Lv0PAK6ACbBl2mBZnnwYFA5aw6GhA3bDxA kaEAoKhOgsCFMe6N0hgVrmLCMebU4OVw =sklc -END PGP SIGNATURE- - To unsubscribe, e-mail:
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 14/11/2010 19:11, André Warnier wrote: Great. For the using JMX clients page, may I recommend : http://code.google.com/p/jmxsh/ which for Java ignorami of my level, is a useful tool to do nifty things in a scripting (non-graphic) mode. I use it only for really simple things, but I have the feeling it can used for much more, such as the recurrent take n threads dumps at regular intervals kind of thing. Folks, This is a Wiki we are discussing. Anyone can edit it. Just create an account and start adding content. The content doesn't need to be perfect. Anything that adds to or improves what is already there - even if the change just adds a TODO - is good. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
André, On 14.11.2010 20:11, André Warnier wrote: For the using JMX clients page, may I recommend : http://code.google.com/p/jmxsh/ which for Java ignorami of my level, is a useful tool to do nifty things in a scripting (non-graphic) mode. I use it only for really simple things, but I have the feeling it can used for much more, such as the recurrent take n threads dumps at regular intervals kind of thing. I'm a bit disappointed. I thought you are a my hammer is Perl guy ;) What about http://search.cpan.org/~roland/jmx4perl/ Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
Rainer Jung wrote: André, On 14.11.2010 20:11, André Warnier wrote: For the using JMX clients page, may I recommend : http://code.google.com/p/jmxsh/ which for Java ignorami of my level, is a useful tool to do nifty things in a scripting (non-graphic) mode. I use it only for really simple things, but I have the feeling it can used for much more, such as the recurrent take n threads dumps at regular intervals kind of thing. I'm a bit disappointed. I thought you are a my hammer is Perl guy ;) What about http://search.cpan.org/~roland/jmx4perl/ Meine Güte, how right you are ! I did not even know this mighty module existed ! Digging into Tomcat and Java from Perl, I love the idea ! And like most CPAN modules, it seems to have a good documentation, well-worth reading even if only to understand JMX. Thanks for the pointer ! - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Brian, On 11/14/2010 1:30 AM, Brian wrote: It seems that I solved my problems! So far, these are my conclutions: - Very often, when I restart/redeploy my app, some garbage is left in the memory. I don't know why, given that my code closes everything (relationships with the database, etc), unloads the JDBC drivers, etc. Now I'm restarting Tomcat very often, instead of just restarting/deploying my app. The above is certainly compounding the issues below... - The Tag bodies made some buffers to grow to huge objects. I have told Tomcat to decrease those buffers if they get bigger than the standard size (512 bytes), and now the problem seems to be gone! I wonder why isn't that LIMIT=true directive a standard. I bet millions of sites are having the same problem, and they don't even imagine what a memory profiler is, and how this can be happening. This problem was swallowing hundreds of MB! Yes: hundreds of MBs of buffers for each webapp instance that is not cleanly undeployed can certainly bust your heap. I'm not entirely sure how Tomcat expects to free all those buffers (or simply relies on GC), but it's certainly possible that retaining some reference to a webapp-loaded object ends up keeping those buffers around forever, unable to free them. That isn't a standard option because in (most?) cases where the buffers are being constantly used, the performance increase is significant. Since your webapp is both misbehaving and being re-loaded often, you must use this workaround. I don't mean to say that you are doing anything wrong, but our production webapps aren't undeployed for months at a time -- between releases. What is it that requires you to redeploy your webapp so often? - I configured the Context so it wont use a cache for the static pages. At least for now, I made this to all the contexts. Maybe I will restore this capability to just my java app that doesn't have more than a few static resources, and keep it disabled for the 20 WARs full of static pages. This problem was also swallowing hundreds of MB! Again, these caches might expect to die with the rest of the webapp. If they don't you'll certainly exhaust your heap after a couple of reloads. I believe Chuck and Mark have both suggested that you simply fix your webapp's leaks (as reported at un-deploy) or post the warning messages to allow us to help you fix those. If you fix your resource leaks, you may be able to restore the default value for LIMIT_BUFFER and thereby restore performance of your JSPs. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzh5nsACgkQ9CaO5/Lv0PCqYwCgl6Op7gScXfTvzfupBQIQ/pPH rOUAoJwK7794A/SpbHEW3JQ8k5U1Fv36 =NajF -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Hi Chris, -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Monday, November 15, 2010 09:04 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Brian, On 11/14/2010 1:30 AM, Brian wrote: It seems that I solved my problems! So far, these are my conclutions: - Very often, when I restart/redeploy my app, some garbage is left in the memory. I don't know why, given that my code closes everything (relationships with the database, etc), unloads the JDBC drivers, etc. Now I'm restarting Tomcat very often, instead of just restarting/deploying my app. The above is certainly compounding the issues below... - The Tag bodies made some buffers to grow to huge objects. I have told Tomcat to decrease those buffers if they get bigger than the standard size (512 bytes), and now the problem seems to be gone! I wonder why isn't that LIMIT=true directive a standard. I bet millions of sites are having the same problem, and they don't even imagine what a memory profiler is, and how this can be happening. This problem was swallowing hundreds of MB! Yes: hundreds of MBs of buffers for each webapp instance that is not cleanly undeployed can certainly bust your heap. I'm not entirely sure how Tomcat expects to free all those buffers (or simply relies on GC), but it's certainly possible that retaining some reference to a webapp-loaded object ends up keeping those buffers around forever, unable to free them. mmm OK, this is what I have understood so far: - The tag body buffers are stored in a growing array of chars. There is a pointer that know which of the chars is the last one being used. Initially, they are 512 chars, but if a bigger buffer is needed, the quantity of chars increases. But it never decreases. So if at one instant 1 million of chars are needed, the array will grow for that, but will never shrink even if the next use of the buffer is for a 10 characters value (I'm assuming that there is a pool of buffers to reuse, that the buffers are part of a pool). - Some some reason (I don't know why), sometimes Tomcat thinks that needs a huge buffer, so it makes the array to increase to millions of chars. I have no such big pages in my site, so I can't understand how can that happen, but it does. Then, the buffer stays with millions of chars, so the object uses millions of bytes. Do you think a leak has something to do with that? - I think Tomcat doesn't even expect to free those buffers as you say, but use them again and again instead, cause I THINK (correct me if I'm wrong) that they belong to a pool. So they just say like that forever. - Now that I set the flag, everytime Tomcat decides to empty the content of the buffer so it will be used again (emtpy=set the lastUsedCharPointer to cero), it sets it to a new array of 512 bytes if it was bigger that that. I don't know if this issue is retaled to the other one (leaks at undeployment). That isn't a standard option because in (most?) cases where the buffers are being constantly used, the performance increase is significant. Since your webapp is both misbehaving and being re-loaded often, you must use this workaround. Well, I don't know how can Tomcat think that a 8MB buffer is needed in my site for a page. That is the source of the problem. But even if its my fault somehow (leaks in my code), or just the planets aligned so this will happen, I think Tomcat should always think If the buffer got too big when I clear it, I will make it again to be 512 bytes. That just means creating a new array of chars. How much cost would that be? I don't mean to say that you are doing anything wrong, but our production webapps aren't undeployed for months at a time -- between releases. What is it that requires you to redeploy your webapp so often? I'm constantly developing, improving and correcting my site. - I configured the Context so it wont use a cache for the static pages. At least for now, I made this to all the contexts. Maybe I will restore this capability to just my java app that doesn't have more than a few static resources, and keep it disabled for the 20 WARs full of static pages. This problem was also swallowing hundreds of MB! Again, these caches might expect to die with the rest of the webapp. If they don't you'll certainly exhaust your heap after a couple of reloads. Well, given that I have about 20 WARs, these caches were using about 200MB of ram. Even if my app never leaks when redeploying and doesn't degenerate, or even if I would never stop my app, I don't like to spend 200MB in these caches. I could also limit the size of the caches, but I started trying to disable them, and my site still runs pretty fast without these caches now. I believe Chuck and Mark have both suggested that you simply fix your webapp's leaks
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? sometimes Tomcat thinks that needs a huge buffer, so it makes the array to increase to millions of chars. I have no such big pages in my site This is the crux of the problem - you apparently *do* have something that requires such a large buffer. Don't know if it's nested JSPs, some kind of recursive include, or ???. Unfortunately, I'm not aware of any existing mechanism in Jasper that will log these exceptional allocations, so someone would have to put in some new logging code to catch the situation. The method that does this is reAllocBuff() at the end of org.apache.jasper.runtime.BodyContentImpl; the current algorithm usually just doubles the size of the buffer when the size of the current one is exceeded. It would be easy just to add a simple logging call (or even a print statement, temporarily) that includes a stack trace when some size threshold is exceeded. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Hi Chuck, Now I think you must be right.It is not impossible that one of my pages is huge. They are dynamically created, so something could be wrong in my programming to deliver a huge page for some requests. Isn't there a log of all the requests? I have one running Maybe the log could show the size of the responses if that is possible, I could see what is the URL of the gigantic page and easily find the reason I will check that -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Monday, November 15, 2010 10:38 PM To: Tomcat Users List Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? sometimes Tomcat thinks that needs a huge buffer, so it makes the array to increase to millions of chars. I have no such big pages in my site This is the crux of the problem - you apparently *do* have something that requires such a large buffer. Don't know if it's nested JSPs, some kind of recursive include, or ???. Unfortunately, I'm not aware of any existing mechanism in Jasper that will log these exceptional allocations, so someone would have to put in some new logging code to catch the situation. The method that does this is reAllocBuff() at the end of org.apache.jasper.runtime.BodyContentImpl; the current algorithm usually just doubles the size of the buffer when the size of the current one is exceeded. It would be easy just to add a simple logging call (or even a print statement, temporarily) that includes a stack trace when some size threshold is exceeded. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? Isn't there a log of all the requests? Not by default, but you can enable the AccessLogValve in server.xml, that will display the request URI and the response size. However, if you have trimSpaces on for the JSP servlet, the size shown in the log will be the one after white space is removed - and odds are that it's white space eating up the buffer. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Well, I am using the valve indeed! I will check if it is showing the size of the responses. I guess I can find the huge pages easily there. I'm not using the trimSpaces flag. If I find it, I wont need to limit the buffer. That would be more elegant than having to do it. :-) But the other issue is that I still have to find the reason of the leak. I'm exploring YourKit for that. I guess I will find how to do it soon. -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Tuesday, November 16, 2010 12:00 AM To: Tomcat Users List Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? Isn't there a log of all the requests? Not by default, but you can enable the AccessLogValve in server.xml, that will display the request URI and the response size. However, if you have trimSpaces on for the JSP servlet, the size shown in the log will be the one after white space is removed - and odds are that it's white space eating up the buffer. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 13/11/2010 23:10, Leon Rosenberg wrote: On Sat, Nov 13, 2010 at 10:19 PM, Pid p...@pidster.com wrote: On 12/11/2010 21:27, Leon Rosenberg wrote: P.S. I have a small tool that creates a diff of two subsequent histograms, i can share it if you need it. Post it to the wiki, perhaps? Which page would fit best? Leon If there isn't one already, we could make a new page about diagnostic techniques. I've been meaning to add something about JConsole VisualVM. p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 14 November 2010 16:55, Pid p...@pidster.com wrote: On 13/11/2010 23:10, Leon Rosenberg wrote: On Sat, Nov 13, 2010 at 10:19 PM, Pid p...@pidster.com wrote: On 12/11/2010 21:27, Leon Rosenberg wrote: P.S. I have a small tool that creates a diff of two subsequent histograms, i can share it if you need it. Post it to the wiki, perhaps? Which page would fit best? Leon If there isn't one already, we could make a new page about diagnostic techniques. I've been meaning to add something about JConsole VisualVM. p I'll second that request for a diagnostic tools page. Have been playing with VisualVM just today. Jon -- --- Jon Mercer DirectorAchean Limited http://www.achean.com http://uk.linkedin.com/in/jonmercer ---
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
Jon Mercer wrote: On 14 November 2010 16:55, Pid p...@pidster.com wrote: On 13/11/2010 23:10, Leon Rosenberg wrote: On Sat, Nov 13, 2010 at 10:19 PM, Pid p...@pidster.com wrote: On 12/11/2010 21:27, Leon Rosenberg wrote: P.S. I have a small tool that creates a diff of two subsequent histograms, i can share it if you need it. Post it to the wiki, perhaps? Which page would fit best? Leon If there isn't one already, we could make a new page about diagnostic techniques. I've been meaning to add something about JConsole VisualVM. I'll second that request for a diagnostic tools page. Have been playing with VisualVM just today. I'll third it. May I even additionally suggest, if it does not exist already, a separate section for diagnostics in the WiKi ? Maybe organised into one article per tool, and then separate articles à la How do I diagnose memory leaks, How do I check how much memory my application uses and How do I do something about it ? etc.. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
2010/11/14 André Warnier a...@ice-sa.com: Jon Mercer wrote: On 14 November 2010 16:55, Pid p...@pidster.com wrote: On 13/11/2010 23:10, Leon Rosenberg wrote: On Sat, Nov 13, 2010 at 10:19 PM, Pid p...@pidster.com wrote: On 12/11/2010 21:27, Leon Rosenberg wrote: P.S. I have a small tool that creates a diff of two subsequent histograms, i can share it if you need it. Post it to the wiki, perhaps? Which page would fit best? Leon If there isn't one already, we could make a new page about diagnostic techniques. I've been meaning to add something about JConsole VisualVM. I'll second that request for a diagnostic tools page. Have been playing with VisualVM just today. I'll third it. May I even additionally suggest, if it does not exist already, a separate section for diagnostics in the WiKi ? Maybe organised into one article per tool, and then separate articles à la How do I diagnose memory leaks, How do I check how much memory my application uses and How do I do something about it ? etc.. I thought about creating a Troubleshooting section in the wiki, (or maybe call it Troubleshooting and Diagnostics now), and collect there pages with various recipes where to look for information and what to do to. Stating with where to look for TC version, and where the TC logs are, and what to do with hung threads (-dumps), etc. Another approach could be to reorganize the list in http://wiki.apache.org/tomcat/HowTo into several sections (to make it more readable), and add HowTo/xxx pages from there. http://wiki.apache.org/tomcat/FAQ http://wiki.apache.org/tomcat/HowTo Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 14/11/2010 17:37, Konstantin Kolinko wrote: 2010/11/14 André Warnier a...@ice-sa.com: Jon Mercer wrote: On 14 November 2010 16:55, Pid p...@pidster.com wrote: On 13/11/2010 23:10, Leon Rosenberg wrote: On Sat, Nov 13, 2010 at 10:19 PM, Pid p...@pidster.com wrote: On 12/11/2010 21:27, Leon Rosenberg wrote: P.S. I have a small tool that creates a diff of two subsequent histograms, i can share it if you need it. Post it to the wiki, perhaps? Which page would fit best? Leon If there isn't one already, we could make a new page about diagnostic techniques. I've been meaning to add something about JConsole VisualVM. I'll second that request for a diagnostic tools page. Have been playing with VisualVM just today. I'll third it. May I even additionally suggest, if it does not exist already, a separate section for diagnostics in the WiKi ? Maybe organised into one article per tool, and then separate articles à la How do I diagnose memory leaks, How do I check how much memory my application uses and How do I do something about it ? etc.. I thought about creating a Troubleshooting section in the wiki, (or maybe call it Troubleshooting and Diagnostics now), and collect there pages with various recipes where to look for information and what to do to. Stating with where to look for TC version, and where the TC logs are, and what to do with hung threads (-dumps), etc. Another approach could be to reorganize the list in http://wiki.apache.org/tomcat/HowTo into several sections (to make it more readable), and add HowTo/xxx pages from there. http://wiki.apache.org/tomcat/FAQ http://wiki.apache.org/tomcat/HowTo Here's a start: http://wiki.apache.org/tomcat/FAQ/Troubleshooting_and_Diagnostics p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
Pid wrote: On 14/11/2010 17:37, Konstantin Kolinko wrote: 2010/11/14 André Warnier a...@ice-sa.com: Jon Mercer wrote: On 14 November 2010 16:55, Pid p...@pidster.com wrote: On 13/11/2010 23:10, Leon Rosenberg wrote: On Sat, Nov 13, 2010 at 10:19 PM, Pid p...@pidster.com wrote: On 12/11/2010 21:27, Leon Rosenberg wrote: P.S. I have a small tool that creates a diff of two subsequent histograms, i can share it if you need it. Post it to the wiki, perhaps? Which page would fit best? Leon If there isn't one already, we could make a new page about diagnostic techniques. I've been meaning to add something about JConsole VisualVM. I'll second that request for a diagnostic tools page. Have been playing with VisualVM just today. I'll third it. May I even additionally suggest, if it does not exist already, a separate section for diagnostics in the WiKi ? Maybe organised into one article per tool, and then separate articles à la How do I diagnose memory leaks, How do I check how much memory my application uses and How do I do something about it ? etc.. I thought about creating a Troubleshooting section in the wiki, (or maybe call it Troubleshooting and Diagnostics now), and collect there pages with various recipes where to look for information and what to do to. Stating with where to look for TC version, and where the TC logs are, and what to do with hung threads (-dumps), etc. Another approach could be to reorganize the list in http://wiki.apache.org/tomcat/HowTo into several sections (to make it more readable), and add HowTo/xxx pages from there. http://wiki.apache.org/tomcat/FAQ http://wiki.apache.org/tomcat/HowTo Here's a start: http://wiki.apache.org/tomcat/FAQ/Troubleshooting_and_Diagnostics Great. For the using JMX clients page, may I recommend : http://code.google.com/p/jmxsh/ which for Java ignorami of my level, is a useful tool to do nifty things in a scripting (non-graphic) mode. I use it only for really simple things, but I have the feeling it can used for much more, such as the recurrent take n threads dumps at regular intervals kind of thing. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 12/11/2010 20:35, Brian wrote: Ok, I will do that now! I have taken another snapshot of the JVM a few minutes ago. Now I also see that 160MB are being used by org.apache.jasper.runtime.BodyContextImpl. This contains images of my DYNAMIC pages! That is sort of what I'd expect. A little background: Tag bodies have to be buffered. Jasper (Tomcat's JSP engine) uses a pool of buffers. Tag bodies are expected to be small. The buffer grows (but does not shrink) if the body is large. If you have a lot of tags that have large bodies then you can see an increase in memory usage in this area. To control this see org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER on http://tomcat.apache.org/tomcat-6.0-doc/config/systemprops.html HTH, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 12/11/2010 20:44, Christopher Schultz wrote: In fact, that's a good test: search your memory snapshot for instances of WebappClassLoader. There's a boolean in each one (active I think) It is started. that tells you if it's active. Force a couple of full GCs, then see how many of them are still around. If you have more than 1+webapp count (I think you get one for free plus one for each webapp deployed) then the old, undeployed webapps are still sitting around in memory. Nope, there should be one and only one for each deployed webapp. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? the Eden Space is barely used (10MB right now). The survivor space is even less used (1MB right now). An object is normally created in Eden and will migrate to survivor if still reachable during the next minor GC. In most applications, the vast majority of objects become unreachable very quickly, and never make it to a survivor space. But the Tenured Gen space has 120MB right now! In fact, when my JVM start to eat houndreds of MB, most of that goes to the Tenured Gen. Once the survivor space fills up, long-lived objects migrate to tenured, where they stay until the application discards them. Very large objects may be initially allocated in tenured if they won't fit in Eden. Anything you see in tenured has either been around for quite some time or exceeds the Eden allocation threshold. The dead objects in tenured space won't be cleaned out until a major GC occurs; the JVM tries to minimize the number of those since they take quite a bit more time than a minor GC. You can force a major GC with the JConsole button. The perm gen is using 22MB right now, which is not a lot so I guess it is normal. A one-time look at the size of the PermGen isn't interesting; you need to see whether it increases over time, especially after restarting a webapp. If it does increase after a restart, that means the old instance of the webapp is still hanging around. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 12/11/2010 21:27, Leon Rosenberg wrote: P.S. I have a small tool that creates a diff of two subsequent histograms, i can share it if you need it. Post it to the wiki, perhaps? p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 13/11/2010 01:13, Brian wrote: I had noticed some warnings about that indeed. What might those have been? Posting the information about what you found will help others understand the process you've been through, whether their problem is similar to yours and if a similar solution applies. p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On Sat, Nov 13, 2010 at 10:19 PM, Pid p...@pidster.com wrote: On 12/11/2010 21:27, Leon Rosenberg wrote: P.S. I have a small tool that creates a diff of two subsequent histograms, i can share it if you need it. Post it to the wiki, perhaps? Which page would fit best? Leon p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Hi Leon, Thanks for your suggestion and sorry for responding so late. I had to take a break from this issue at least for a day. I had been working on this for 3 days in a row and barely sleeping at night, because of the crashes. As soon as I saw my installation not crashing, I needed to take a rest. I'm already using www.YourKit.com for now. It is not free, I guess I will not buy it finally if I totally solve my problem. But for now, while in the trial period, I'm using it and it is great! But as soon as it ends, I will definitely try Jmap/Jhat as you suggested. -Original Message- From: Leon Rosenberg [mailto:rosenberg.l...@gmail.com] Sent: Friday, November 12, 2010 04:27 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? Hello Brian, maybe I missed half of the communication, but from the other half I got the feeling that you are shooting in the dark. Heap dumps are hard to decipher especially if the internals seems to be unknown ;-) When hunting a memory leak I setup a cron job that performs the same task once an hour: jmap -heap:live pid file-with-timestamp.heap jmap -histo:live pid file- with-timestamp.histo the jmap histogramm contains all objects in your vm and their cumulated space. by comparing two of them taken in 30 or 60 minutes you can determine which objects are actually increasing numbers or size. With that knowledge analyzing heap dumps can be performed much faster and easier. Keep in mind that analyzing mem leaks that lead to outofmemory after the oome occured is twice as hard as shortly before . regards Leon P.S. I have a small tool that creates a diff of two subsequent histograms, i can share it if you need it. P.P.S. jmap is a standart java tool. Another standart java tool - jhat can theoretically analyze a heap dump based on a baseline heapdump taken previously. On Fri, Nov 12, 2010 at 9:44 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? Now I also see that 160MB are being used by org.apache.jasper.runtime.BodyContextImpl. There are a couple of system properties you can set to control this: org.apache.jasper.runtime.JspFactoryImpl.USE_POOL org.apache.jasper.runtime.JspFactoryImpl.POOL_SIZE Look here for the doc: http://tomcat.apache.org/tomcat-6.0-doc/config/systemprops.html - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Hi Mark, This is interesting. I got a snapshot of the memory. I found just about 91 instances of this class (org.apache.jasper.runtime.BodyContentImpl). Just 91, so I guess the quantity of buffers at the same time is small, maybe they belong to a pool and the maximum quantity of objects in the pool is 100? I'm just guessing. Well, one of them was as big as 18MB! The second one almost the same about the first 10 of them where HUGE. Then the rest where more reasonable, between 1 and 50KB. How can a page in my site be as big as 18MB? It definitely can't happen. But for some reason, the buffer was that big in a certain point of time. I inspected the content of them, and they were not huge inside, just a few KB of HTML code inside and the rest was spaces or some invisible character. I mean, these instances of the class contained a huge array of chars, buf only the first hundreds/thousands of them contained HTML code, but given that the busfferSize variable contained a big value (18 millons), the object thought it had a huge full buffer containing real values. I guess something wrong happened, maybe an exception ocurred or I undeployed my app at that very moment (or something else went wrong) and this huge buffers got wrongly full of dummy characters and then stayed in the limbo, until the GB would delete them? Or now that I think more about it, they stayed in the memory to be reused again, being objects in a pool, their nextChar variable (a pointer) was reset to a small value, but their hugely increased internal array of chars was going to stay as big as the biggest they were... like you said. I will set the LIMIT_BUFFER value now, I guess that will solve it as you said. When the buffer gets cleared after it is being used every time, and the LIMIT value =true, the buffer will shrink to 512 bytes again, huh? That will be nice. :-) -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Saturday, November 13, 2010 07:21 AM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 12/11/2010 20:35, Brian wrote: Ok, I will do that now! I have taken another snapshot of the JVM a few minutes ago. Now I also see that 160MB are being used by org.apache.jasper.runtime.BodyContextImpl. This contains images of my DYNAMIC pages! That is sort of what I'd expect. A little background: Tag bodies have to be buffered. Jasper (Tomcat's JSP engine) uses a pool of buffers. Tag bodies are expected to be small. The buffer grows (but does not shrink) if the body is large. If you have a lot of tags that have large bodies then you can see an increase in memory usage in this area. To control this see org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER on http://tomcat.apache.org/tomcat-6.0-doc/config/systemprops.html HTH, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Hi Chuck, Thanks for your explanations. My PermGen seems to be normal, low and steady. It seems that I solved my problems! So far, these are my conclutions: - Very often, when I restart/redeploy my app, some garbage is left in the memory. I don't know why, given that my code closes everything (relationships with the database, etc), unloads the JDBC drivers, etc. Now I'm restarting Tomcat very often, instead of just restarting/deploying my app. - The Tag bodies made some buffers to grow to huge objects. I have told Tomcat to decrease those buffers if they get bigger than the standard size (512 bytes), and now the problem seems to be gone! I wonder why isn't that LIMIT=true directive a standard. I bet millions of sites are having the same problem, and they don't even imagine what a memory profiler is, and how this can be happening. This problem was swallowing hundreds of MB! - I configured the Context so it wont use a cache for the static pages. At least for now, I made this to all the contexts. Maybe I will restore this capability to just my java app that doesn't have more than a few static resources, and keep it disabled for the 20 WARs full of static pages. This problem was also swallowing hundreds of MB! Brian -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Saturday, November 13, 2010 11:03 AM To: Tomcat Users List Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? the Eden Space is barely used (10MB right now). The survivor space is even less used (1MB right now). An object is normally created in Eden and will migrate to survivor if still reachable during the next minor GC. In most applications, the vast majority of objects become unreachable very quickly, and never make it to a survivor space. But the Tenured Gen space has 120MB right now! In fact, when my JVM start to eat houndreds of MB, most of that goes to the Tenured Gen. Once the survivor space fills up, long-lived objects migrate to tenured, where they stay until the application discards them. Very large objects may be initially allocated in tenured if they won't fit in Eden. Anything you see in tenured has either been around for quite some time or exceeds the Eden allocation threshold. The dead objects in tenured space won't be cleaned out until a major GC occurs; the JVM tries to minimize the number of those since they take quite a bit more time than a minor GC. You can force a major GC with the JConsole button. The perm gen is using 22MB right now, which is not a lot so I guess it is normal. A one-time look at the size of the PermGen isn't interesting; you need to see whether it increases over time, especially after restarting a webapp. If it does increase after a restart, that means the old instance of the webapp is still hanging around. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 12/11/2010 05:54, Brian wrote: Hi Pid, I did it, but shows no results. Anyway, it was nice to learn about Jconsole. Now I wonder what is the tool I could use to inspect the objets inside my app, and see which ones are using all the memory. Try VisualVM, another JDK6 tool, but a more recent version is available: http://visualvm.dev.java.net/ There are extra plugins available from the Tools menu. You can use profiling on local JVM processes to see which classes are using memory, CPU time. Or take a series of heap dump and import, examine them. p -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Thursday, November 11, 2010 03:06 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 11/11/2010 18:54, Brian wrote: I don't think my app is taking all this RAM, because when I restart it, the RAM usage doesn't go down. It does only if I restart Tomcat itself, instead of my app running there. Yes, this is a classic sign of a problem with the app. Reboot Tomcat, restart your app a couple of times (this bit is important). Connect to the Tomcat instance using JConsole, navigate the MBeans, to Catalina Hosts (your hostname), then select the Operations tab, under which you'll see a button called findReloadContextMemoryLeaks. Push the button. It will return a list of app names if Tomcat can detect ones with memory leaks. NB No results doesn't necessarily mean your app isn't leaking. p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Great advice, thanks I'm using it right now, and I'm also using www.yourkit.com as a trial. My problem is in the Tenured/Gen area. There is where most of the RAM is getting used. Now I need to fin out what Tenured/Gen is about -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Friday, November 12, 2010 03:49 AM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 12/11/2010 05:54, Brian wrote: Hi Pid, I did it, but shows no results. Anyway, it was nice to learn about Jconsole. Now I wonder what is the tool I could use to inspect the objets inside my app, and see which ones are using all the memory. Try VisualVM, another JDK6 tool, but a more recent version is available: http://visualvm.dev.java.net/ There are extra plugins available from the Tools menu. You can use profiling on local JVM processes to see which classes are using memory, CPU time. Or take a series of heap dump and import, examine them. p -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Thursday, November 11, 2010 03:06 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 11/11/2010 18:54, Brian wrote: I don't think my app is taking all this RAM, because when I restart it, the RAM usage doesn't go down. It does only if I restart Tomcat itself, instead of my app running there. Yes, this is a classic sign of a problem with the app. Reboot Tomcat, restart your app a couple of times (this bit is important). Connect to the Tomcat instance using JConsole, navigate the MBeans, to Catalina Hosts (your hostname), then select the Operations tab, under which you'll see a button called findReloadContextMemoryLeaks. Push the button. It will return a list of app names if Tomcat can detect ones with memory leaks. NB No results doesn't necessarily mean your app isn't leaking. p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
I found the problem and this time it wasn't my fault! :-) I used a profiler (www.yourkit.com) and took a snapshot of all the objects in the JVM. I found that tons of RAM is being used by images of my delivered http responses. I mean, I have thousands of static HTML pages in my site. It seems that Tomcat is saving their content in the memory, as a cache, in thousands of objects. That was the hardest part, to find out that. Now I wonder where do I configure Tomcat to stop doing this, or at least to reduce the cache. -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Thursday, November 11, 2010 02:11 PM To: Tomcat Users List Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: Tomcat 6.0.29 using more and more RAM until it collapses? I don't think my app is taking all this RAM It is. when I restart it, the RAM usage doesn't go down. Classic symptom of a webapp leaking memory due to holding onto It seems like if Tomcat is leaving garbage in the JVM or something like that. Nope - it's your webapp. Does anybody know what should I do to solve this? Fix your webapp(s). Start with a heap profiler, and look to see what's consuming the space. The problem may also include memory leaks outside of the heap, such as failing to close files, or starting and forgetting about auxiliary threads. Start reading here: http://wiki.apache.org/tomcat/FAQ/Memory http://wiki.apache.org/tomcat/OutOfMemory http://wiki.apache.org/tomcat/MemoryLeakProtection - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 12/11/2010 18:13, Brian wrote: I found the problem and this time it wasn't my fault! :-) I wouldn't be so sure about that just yet. I used a profiler (www.yourkit.com) and took a snapshot of all the objects in the JVM. I found that tons of RAM is being used by images of my delivered http responses. I mean, I have thousands of static HTML pages in my site. It seems that Tomcat is saving their content in the memory, as a cache, in thousands of objects. Tomcat's static content cache is 10MB per web application by default. I suspect something else is holding on to those references. Where to the GC roots trace to? That was the hardest part, to find out that. Now I wonder where do I configure Tomcat to stop doing this, or at least to reduce the cache. You need to confirm what is holding on to those references first. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Hi Mark, Besides using Tomcat to serve my app (which itself serves about 600,000 diferent URLs), I also use it to serve static pages (HTML pages stored in the hard disk). I don't know if Tomcat is caching my dinamically generated pages (maybe it is), but for now I would love to stop the cache that stores the static ones. Those static pages are in their own WARs, so they have nothing to do with the java code in my app. Where do I configure Tomcat to stop caching the static pages in memory? This is definitely my problem, I can see clearly the pages in my memory snapshot, and when I let Google crawl my site with more speed, my site (and Tomcat itself, I guess) crashes sooner. -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Friday, November 12, 2010 01:19 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 12/11/2010 18:13, Brian wrote: I found the problem and this time it wasn't my fault! :-) I wouldn't be so sure about that just yet. I used a profiler (www.yourkit.com) and took a snapshot of all the objects in the JVM. I found that tons of RAM is being used by images of my delivered http responses. I mean, I have thousands of static HTML pages in my site. It seems that Tomcat is saving their content in the memory, as a cache, in thousands of objects. Tomcat's static content cache is 10MB per web application by default. I suspect something else is holding on to those references. Where to the GC roots trace to? That was the hardest part, to find out that. Now I wonder where do I configure Tomcat to stop doing this, or at least to reduce the cache. You need to confirm what is holding on to those references first. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 12/11/2010 18:30, Brian wrote: Hi Mark, Besides using Tomcat to serve my app (which itself serves about 600,000 diferent URLs), I also use it to serve static pages (HTML pages stored in the hard disk). I don't know if Tomcat is caching my dinamically generated pages (maybe it is), but for now I would love to stop the cache that stores the static ones. Those static pages are in their own WARs, so they have nothing to do with the java code in my app. Where do I configure Tomcat to stop caching the static pages in memory? This is definitely my problem, I can see clearly the pages in my memory snapshot, and when I let Google crawl my site with more speed, my site (and Tomcat itself, I guess) crashes sooner. You are missing the point. It is highly unlikely the 10MB per webapp cache is causing problems given that you previously states that there are only two WAR files here. So I'll ask my question again: Where to the GC roots for those objects trace to? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Well, maybe you can help me to make an interpretation. I'm using Yourkit, and this this what I see. The most RAM is used by byte arrays. If I choose this entry in the list of classes, below you will see a list of objects. If you choose the first one and check the content (400kb), it is the image of a static HTML page. I really don't know how to find out to which classes do these bytes belong to, or their relationship with the garbage collector, but I can see that they use a lot of RAM. -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Friday, November 12, 2010 01:38 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 12/11/2010 18:30, Brian wrote: Hi Mark, Besides using Tomcat to serve my app (which itself serves about 600,000 diferent URLs), I also use it to serve static pages (HTML pages stored in the hard disk). I don't know if Tomcat is caching my dinamically generated pages (maybe it is), but for now I would love to stop the cache that stores the static ones. Those static pages are in their own WARs, so they have nothing to do with the java code in my app. Where do I configure Tomcat to stop caching the static pages in memory? This is definitely my problem, I can see clearly the pages in my memory snapshot, and when I let Google crawl my site with more speed, my site (and Tomcat itself, I guess) crashes sooner. You are missing the point. It is highly unlikely the 10MB per webapp cache is causing problems given that you previously states that there are only two WAR files here. So I'll ask my question again: Where to the GC roots for those objects trace to? Mark - To unsubscribe, e-mail: mailto:users-unsubscr...@tomcat.apache.org users-unsubscr...@tomcat.apache.org For additional commands, e-mail: mailto:users-h...@tomcat.apache.org users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 12/11/2010 18:59, Brian wrote: Well, maybe you can help me to make an interpretation. I'm using Yourkit, and this this what I see. The most RAM is used by byte arrays. If I choose this entry in the list of classes, below you will see a list of objects. If you choose the first one and check the content (400kb), it is the image of a static HTML page. I really don't know how to find out to which classes do these bytes belong to, or their relationship with the garbage collector, but I can see that they use a lot of RAM. Look in the YourKit docs for how to trace the GC roots. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? I'm using Yourkit, and this this what I see. But we can't, since the mailing list strips attachments. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Oh, I think I found what you were asking, the path to the GC root: org.apache.naming.resources.FileDirContext$FileResource org.apache.naming.resources.CacheEntry org.apache.naming.resources.CacheEntry[XXX] org.apache.naming.resources.ResourceCache org.apache.naming.resources.ProxyDirContext etc etc etc -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Friday, November 12, 2010 01:38 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 12/11/2010 18:30, Brian wrote: Hi Mark, Besides using Tomcat to serve my app (which itself serves about 600,000 diferent URLs), I also use it to serve static pages (HTML pages stored in the hard disk). I don't know if Tomcat is caching my dinamically generated pages (maybe it is), but for now I would love to stop the cache that stores the static ones. Those static pages are in their own WARs, so they have nothing to do with the java code in my app. Where do I configure Tomcat to stop caching the static pages in memory? This is definitely my problem, I can see clearly the pages in my memory snapshot, and when I let Google crawl my site with more speed, my site (and Tomcat itself, I guess) crashes sooner. You are missing the point. It is highly unlikely the 10MB per webapp cache is causing problems given that you previously states that there are only two WAR files here. So I'll ask my question again: Where to the GC roots for those objects trace to? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Brian, On 11/12/2010 11:19 AM, Brian wrote: Great advice, thanks I'm using it right now, and I'm also using www.yourkit.com as a trial. My problem is in the Tenured/Gen area. There is where most of the RAM is getting used. Now I need to fin out what Tenured/Gen is about Would you mind taking a look back at my message from yesterday? I asked some questions that might be able to significantly reduce the effort you expend during your search. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzdkuUACgkQ9CaO5/Lv0PAaSgCfazR0EQlmQ8HgreOnxXJHjDlq uXoAnjMQKBEEer2Q5DYIhilqcuwdyTzk =VpG0 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 12/11/2010 19:12, Brian wrote: Oh, I think I found what you were asking, the path to the GC root: org.apache.naming.resources.FileDirContext$FileResource org.apache.naming.resources.CacheEntry org.apache.naming.resources.CacheEntry[XXX] org.apache.naming.resources.ResourceCache org.apache.naming.resources.ProxyDirContext etc etc etc OK, that is indeed Tomcat's static file cache. Given you only have two WARs and the default cache size is 10MB per WAR I am struggling to see how this could be causing an OOME. Anyway, the settings for this are on the Context element. The simplest way to disable static file caching everywhere is to add the following to the Context element in CATALINA_BASE/conf/context.xml cachingAllowed=false Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Brian, On 11/12/2010 2:12 PM, Brian wrote: Oh, I think I found what you were asking, the path to the GC root: org.apache.naming.resources.FileDirContext$FileResource org.apache.naming.resources.CacheEntry org.apache.naming.resources.CacheEntry[XXX] org.apache.naming.resources.ResourceCache org.apache.naming.resources.ProxyDirContext etc etc etc That etc. is important information. Keep going. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzdk40ACgkQ9CaO5/Lv0PCbeQCgvQIBv7AHHN4kuQMiZaN17hpu oSoAnRK4VoH++sZS4/13dqLQmUXO95ki =Ev1I -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 11/12/2010 2:12 PM, Brian wrote: Oh, I think I found what you were asking, the path to the GC root: org.apache.naming.resources.FileDirContext$FileResource org.apache.naming.resources.CacheEntry org.apache.naming.resources.CacheEntry[XXX] org.apache.naming.resources.ResourceCache org.apache.naming.resources.ProxyDirContext etc etc etc That etc. is important information. Keep going. Also, a given object may be reachable from more than one root, so show them all. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Byte[401494] = {10,10,10,10,.} This
contains the whole 400KB HTML code
binaryContent oforg.apache.naming.resources.FileDirContext$FileResource
resource of org.apache.naming.resources.CacheEntry
[2]of org.apache.naming.resources.CacheEntry[6]
Cache oforg.apache.naming.resources.ResourceCache
Cache of org.apache.naming.resources.ProxyDirContext
Value of java.util.Hashtable$Entry
[20]ofjava.util.Hashtable$Entry[47]
Table of java.util.Hashtable
clBindings oforg.apache.naming.resources.DirContextURLStreamHandler
[275 of java.lang.Object[640]
elementData of java.util.Vector
classes of org.apache.catalina.loader.StandardClassLoader
contextClassLoader of java.lang.Thread [Stack Local, Thread] http-8080-52
native ID: xx
-Original Message-
From: Christopher Schultz [mailto:ch...@christopherschultz.net]
Sent: Friday, November 12, 2010 02:21 PM
To: Tomcat Users List
Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Brian,
On 11/12/2010 2:12 PM, Brian wrote:
Oh, I think I found what you were asking, the path to the GC root:
org.apache.naming.resources.FileDirContext$FileResource
org.apache.naming.resources.CacheEntry
org.apache.naming.resources.CacheEntry[XXX]
org.apache.naming.resources.ResourceCache
org.apache.naming.resources.ProxyDirContext
etc
etc
etc
That etc. is important information. Keep going.
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkzdk40ACgkQ9CaO5/Lv0PCbeQCgvQIBv7AHHN4kuQMiZaN17h
pu
oSoAnRK4VoH++sZS4/13dqLQmUXO95ki
=Ev1I
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Hi Chris, I saved your email and hadn't replied yet. Here are my responses, thanks! -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Thursday, November 11, 2010 02:39 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? Brian, On 11/11/2010 1:54 PM, Brian wrote: I don't think my app is taking all this RAM, because when I restart it, the RAM usage doesn't go down. That doesn't necessarily mean that your webapp isn't using all that heap space: it's very easy for a webapp to do something foolish and cause all of it's used memory to stay active even after a webapp restart. Yes, now I know that. I disagree with Ben's comment about database connections: there are faster ways to bust your heap than leaking database connections. If you /are/ using Tomcat's container-managed DataSource (which I highly recommend), you should enable all of the abandoned resource tracking features to help fix any resource leaks that you may have there. Well, the database relation with my code is not the problem, I'm using a pool and it works great. I don’t know for what else should I use the DataSource. I agree with Chuck's assessment, but there may be some questions we can ask to help lead you down the right path when using tools such as memory profilers. 0. What kind of OOME are you suffering? Please post any messages that you get when your heap busts. Specifically, is this a regular heap exhaustion or a PermGen exhaustion? Good question, but I can't answer that. I think I lost my logs. :-( I will search for them again. But I could use the profiler now and responde whatever you would like to ask about how is it behaving now. 1. Are you seeing any messages in catalina.out when you undeploy of the form your webapp likely has a memory leak? Tomcat 6.0.x has some nice checks on undeploy to see if your webapp is leaking some specific things like threads, etc. Yeah, I have seen those lately. 2. Are you re-starting your webapp a lot while Tomcat remains running? Failure to perform some clean-up during webapp unload can cause your entire set of classes loaded in the old webapp to stay in the PermGen space forever. No, I don’t restart them frequently. But I do redeploy them frequently, in the past I did it deleting the WAR in the directly and uploading a new one, now I'm suing the Tmocat Manager do undeploy and deploy again. 3. Are you using any caching mechanism in your webapp? Perhaps it needs to be tuned. You should probably check out what's in your application scope: you may have things you didn't expect. I do have some caches in my objects. For example, the ir an object called BrandsManager which has methods like getBrands(), getBrand(int id), insertBrand and so on. The getBrands() method goes to the database only the first time, and then creates a LinkedList with the data. But I don't think this kind of caching is the problem, they consume a minimal amount of memory. 4. Are you using HttpSession for anything bulky? It's possible that you are leaking memory with lots of sessions. When a webapp is stopped, though, all session should be purged and if you have session persistence, they should all come back and take up the same amount of memory. This is a long-shot, but low-hanging fruit. No, I don’t store anything big in the sessions. I just store some strings in them. Nothing serious. How do I know if I have session persistence? You can get a lot of mileage out of a tool like jmap which comes with the JDK (and JRE?): you can get a heap histogram and look at things that are taking up a lot of space. If you find that you have several tens of megabytes of foo.bar.Baz classes, then you can look at your app to see where you heavy uses of those classes are. I'm already using yourkit, which I guess is even better. To understand #2 a bit better, see Mark's presentation from this year's ApacheCon NA: http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks- 60mins.pdf I will now!! It's worth a read to understand how simple things like using a logging library can cause PermGen exhaustion after several webapp redeploy operations. Well, I'm using the Tomcat log indeed. And I also saw a lot of memory being used by that! But more memory is being used by the cached static pages, so I guess I should start with that. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Ok, I will do that now! I have taken another snapshot of the JVM a few minutes ago. Now I also see that 160MB are being used by org.apache.jasper.runtime.BodyContextImpl. This contains images of my DYNAMIC pages! This is the path: Org.apache.jasper.runtime.BodyContentIml [1] of org.apache.jasper.runtime.BodyContentImpl[2] Outs of org.apache.jasper.runtime.PageContextImpl [0] of javax.servlet.jsp.PageContext[8] Pool of org.apache.jasper.runtime.JspFactoryImpl$PageContextPool Value of java.lang.ThreadLocal$ThreadLocalMap$Entry [8] of java.lang.ThreadLocal$ThreadLocalMap$Entry[16] Table of java.lang.ThreadLocal$ThreadLocalMap threadLocals of java.lang.Thread [Stack Local, Thread] http-8080-18 native ID: XX -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Friday, November 12, 2010 02:19 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 12/11/2010 19:12, Brian wrote: Oh, I think I found what you were asking, the path to the GC root: org.apache.naming.resources.FileDirContext$FileResource org.apache.naming.resources.CacheEntry org.apache.naming.resources.CacheEntry[XXX] org.apache.naming.resources.ResourceCache org.apache.naming.resources.ProxyDirContext etc etc etc OK, that is indeed Tomcat's static file cache. Given you only have two WARs and the default cache size is 10MB per WAR I am struggling to see how this could be causing an OOME. Anyway, the settings for this are on the Context element. The simplest way to disable static file caching everywhere is to add the following to the Context element in CATALINA_BASE/conf/context.xml cachingAllowed=false Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Brian, On 11/12/2010 3:02 PM, Brian wrote: -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Thursday, November 11, 2010 02:39 PM I agree with Chuck's assessment, but there may be some questions we can ask to help lead you down the right path when using tools such as memory profilers. 0. What kind of OOME are you suffering? Please post any messages that you get when your heap busts. Specifically, is this a regular heap exhaustion or a PermGen exhaustion? Good question, but I can't answer that. I think I lost my logs. :-( I will search for them again. But I could use the profiler now and responde whatever you would like to ask about how is it behaving now. I think you already answered this elsewhere: you're filling-up your Tenured generation and/or PermGen. Keep an eye on PermGen: if it's filling up after your 60 URLs (that's a /lot/ IMO) have all been hit, then you might have a redeployment problem. 1. Are you seeing any messages in catalina.out when you undeploy of the form your webapp likely has a memory leak? Tomcat 6.0.x has some nice checks on undeploy to see if your webapp is leaking some specific things like threads, etc. Yeah, I have seen those lately. Read them, and try to fix them. ;) 2. Are you re-starting your webapp a lot while Tomcat remains running? Failure to perform some clean-up during webapp unload can cause your entire set of classes loaded in the old webapp to stay in the PermGen space forever. No, I don’t restart them frequently. But I do redeploy them frequently, in the past I did it deleting the WAR in the directly and uploading a new one, now I'm suing the Tmocat Manager do undeploy and deploy again. Semantics. You are stopping your webapp and starting it again, with or without an update to the WAR file. It's actually the ungraceful webapp stop that kills you, not the fact that you start up again or whether it's a restart or a redeploy. If your webapp doesn't clean-up, the WebappClassLoader can stay in memory forever. In fact, that's a good test: search your memory snapshot for instances of WebappClassLoader. There's a boolean in each one (active I think) that tells you if it's active. Force a couple of full GCs, then see how many of them are still around. If you have more than 1+webapp count (I think you get one for free plus one for each webapp deployed) then the old, undeployed webapps are still sitting around in memory. If that's the case, then you are filling-up your PermGen with useless java.lang.Class instances. I'm not sure how Tomcat expires it's caches, but it might also be keeping those cached static files around with the WebappClassLoader, too. Double whammy. 3. Are you using any caching mechanism in your webapp? Perhaps it needs to be tuned. You should probably check out what's in your application scope: you may have things you didn't expect. I do have some caches in my objects. For example, the ir an object called BrandsManager which has methods like getBrands(), getBrand(int id), insertBrand and so on. The getBrands() method goes to the database only the first time, and then creates a LinkedList with the data. But I don't think this kind of caching is the problem, they consume a minimal amount of memory. It's always a good idea to double-check. 4. Are you using HttpSession for anything bulky? It's possible that you are leaking memory with lots of sessions. When a webapp is stopped, though, all session should be purged and if you have session persistence, they should all come back and take up the same amount of memory. This is a long-shot, but low-hanging fruit. No, I don’t store anything big in the sessions. I just store some strings in them. Nothing serious.How do I know if I have session persistence? You get it for free if you use the standard manager: it will persist sessions to the disk when Tomcat stops, and re-load them when Tomcat comes back up. This is not session persistence like using a DbStore where all your session data goes to a db or anything like that. It's worth a read to understand how simple things like using a logging library can cause PermGen exhaustion after several webapp redeploy operations. Well, I'm using the Tomcat log indeed. And I also saw a lot of memory being used by that! But more memory is being used by the cached static pages, so I guess I should start with that. Tomcat's log shouldn't be using any noticeable amount of memory. The only static file you mentioned was a 400KiB file, which is only 4% of the default cache size of 10MiB. Are you seeing /lots/ of those files? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzdpycACgkQ9CaO5/Lv0PDvJgCfUHYFH2uP/5gYKrb84Y/N/SoN LZEAn2hjU3bjQqx/6+6OdPVdexd6mkkX =dPtI
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? Now I also see that 160MB are being used by org.apache.jasper.runtime.BodyContextImpl. There are a couple of system properties you can set to control this: org.apache.jasper.runtime.JspFactoryImpl.USE_POOL org.apache.jasper.runtime.JspFactoryImpl.POOL_SIZE Look here for the doc: http://tomcat.apache.org/tomcat-6.0-doc/config/systemprops.html - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
Hello Brian, maybe I missed half of the communication, but from the other half I got the feeling that you are shooting in the dark. Heap dumps are hard to decipher especially if the internals seems to be unknown ;-) When hunting a memory leak I setup a cron job that performs the same task once an hour: jmap -heap:live pid file-with-timestamp.heap jmap -histo:live pid file-with-timestamp.histo the jmap histogramm contains all objects in your vm and their cumulated space. by comparing two of them taken in 30 or 60 minutes you can determine which objects are actually increasing numbers or size. With that knowledge analyzing heap dumps can be performed much faster and easier. Keep in mind that analyzing mem leaks that lead to outofmemory after the oome occured is twice as hard as shortly before . regards Leon P.S. I have a small tool that creates a diff of two subsequent histograms, i can share it if you need it. P.P.S. jmap is a standart java tool. Another standart java tool - jhat can theoretically analyze a heap dump based on a baseline heapdump taken previously. On Fri, Nov 12, 2010 at 9:44 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? Now I also see that 160MB are being used by org.apache.jasper.runtime.BodyContextImpl. There are a couple of system properties you can set to control this: org.apache.jasper.runtime.JspFactoryImpl.USE_POOL org.apache.jasper.runtime.JspFactoryImpl.POOL_SIZE Look here for the doc: http://tomcat.apache.org/tomcat-6.0-doc/config/systemprops.html - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Hi Chris, I will answer everything here: - In my Heap Memory, the Eden Space is barely used (10MB right now). The survivor space is even less used (1MB right now). But the Tenured Gen space has 120MB right now! In fact, when my JVM start to eat houndreds of MB, most of that goes to the Tenured Gen. Why is that? I would like to know. The perm gen is using 22MB right now, which is not a lot so I guess it is normal. - From now on, when I stop an application, I will check if that left a memory leak. I guess I can do that with the new buton on the Tmcat Manager. - I suspect what is the problem when I stop my app: It is a website, so about 30 humans on average are making requests at any time, and the crawlers (specially Googlebot) are doing it as well. Maybe if I stop it when a request was coming, that makes the licking because maybe some class loaders are doing somethin while I'm asking to stop the app. What do you think about this? - I will check the WebappClassLoader issue, thanks for the tip. -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Friday, November 12, 2010 03:44 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Brian, On 11/12/2010 3:02 PM, Brian wrote: -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Thursday, November 11, 2010 02:39 PM I agree with Chuck's assessment, but there may be some questions we can ask to help lead you down the right path when using tools such as memory profilers. 0. What kind of OOME are you suffering? Please post any messages that you get when your heap busts. Specifically, is this a regular heap exhaustion or a PermGen exhaustion? Good question, but I can't answer that. I think I lost my logs. :-( I will search for them again. But I could use the profiler now and responde whatever you would like to ask about how is it behaving now. I think you already answered this elsewhere: you're filling-up your Tenured generation and/or PermGen. Keep an eye on PermGen: if it's filling up after your 60 URLs (that's a /lot/ IMO) have all been hit, then you might have a redeployment problem. 1. Are you seeing any messages in catalina.out when you undeploy of the form your webapp likely has a memory leak? Tomcat 6.0.x has some nice checks on undeploy to see if your webapp is leaking some specific things like threads, etc. Yeah, I have seen those lately. Read them, and try to fix them. ;) 2. Are you re-starting your webapp a lot while Tomcat remains running? Failure to perform some clean-up during webapp unload can cause your entire set of classes loaded in the old webapp to stay in the PermGen space forever. No, I don t restart them frequently. But I do redeploy them frequently, in the past I did it deleting the WAR in the directly and uploading a new one, now I'm suing the Tmocat Manager do undeploy and deploy again. Semantics. You are stopping your webapp and starting it again, with or without an update to the WAR file. It's actually the ungraceful webapp stop that kills you, not the fact that you start up again or whether it's a restart or a redeploy. If your webapp doesn't clean-up, the WebappClassLoader can stay in memory forever. In fact, that's a good test: search your memory snapshot for instances of WebappClassLoader. There's a boolean in each one (active I think) that tells you if it's active. Force a couple of full GCs, then see how many of them are still around. If you have more than 1+webapp count (I think you get one for free plus one for each webapp deployed) then the old, undeployed webapps are still sitting around in memory. If that's the case, then you are filling-up your PermGen with useless java.lang.Class instances. I'm not sure how Tomcat expires it's caches, but it might also be keeping those cached static files around with the WebappClassLoader, too. Double whammy. 3. Are you using any caching mechanism in your webapp? Perhaps it needs to be tuned. You should probably check out what's in your application scope: you may have things you didn't expect. I do have some caches in my objects. For example, the ir an object called BrandsManager which has methods like getBrands(), getBrand(int id), insertBrand and so on. The getBrands() method goes to the database only the first time, and then creates a LinkedList with the data. But I don't think this kind of caching is the problem, they consume a minimal amount of memory. It's always a good idea to double-check. 4. Are you using HttpSession for anything bulky? It's possible that you are leaking memory with lots of sessions. When a webapp is stopped, though, all session should be purged and if you have session persistence
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Hi Mark, Chris, and all the gurus, I haven't applied this change yet, but something good is happening: Right now, my Tomcat installation is running fine! So given that I havent made any changes to my code, I guess my problem is that sometimes when I stop my website, something goes wrong and not al the resources get liberated. I had noticed some warnings about that indeed. So I guess my problem was not the static/dynamic pages cache as I thought when using the profiler, or they way the DBMS is handled, or any other issue in my programming. As you said from the beginning, when you said that the 10MB cache should not be a problem. So now, given that right now the server is runnning fine, I guess I should concentrate on the problem that sometimes happen when stopping my website. -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Friday, November 12, 2010 02:19 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 12/11/2010 19:12, Brian wrote: Oh, I think I found what you were asking, the path to the GC root: org.apache.naming.resources.FileDirContext$FileResource org.apache.naming.resources.CacheEntry org.apache.naming.resources.CacheEntry[XXX] org.apache.naming.resources.ResourceCache org.apache.naming.resources.ProxyDirContext etc etc etc OK, that is indeed Tomcat's static file cache. Given you only have two WARs and the default cache size is 10MB per WAR I am struggling to see how this could be causing an OOME. Anyway, the settings for this are on the Context element. The simplest way to disable static file caching everywhere is to add the following to the Context element in CATALINA_BASE/conf/context.xml cachingAllowed=false Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
The most common cause of this, that I've seen is the failure to properly close all database connections. If you're using the container managed connection pooling, it is possible that restart your app won't free the ram consumed by any orphaned connections. Without seeing everthing you're doing everything else would be a guess. -Ben On Thu, 2010-11-11 at 13:54 -0500, Brian wrote: Hi, In my Linux machine, I'm using the JVM version 1.6.0_11-b03. On top of that, I only run Tomcat 6.0.29. On that Tomcat installation, I'm running a couple of sites, both of them use exactly the same code, actually it is the same WAR. So they are two apps, but we could consider them as one. The problem is that the RAM usage in the server starts to grow day by day, until Tomcat stops (freezes? hungs?) and my two sites stop working. According to the OS (Linux), it is the Tomcat process that is taking up all that amount of RAM. When I restart Tomcat, it starts fine again, but starts to grow in memory usage day by day again, until it crashes. I don't think my app is taking all this RAM, because when I restart it, the RAM usage doesn't go down. It does only if I restart Tomcat itself, instead of my app running there. It seems like if Tomcat is leaving garbage in the JVM or something like that. According to the Tomcat manager application, in the server status page, the JVM total memory value grows all the time. In this very moment, for example, it says this: Free memory: 178.94 MB Total memory: 565.58 MB Max memory: 692.25 MB. The total memory value is the one that starts growing when Tomcat starts. Does anybody know what should I do to solve this? Thanks! - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: Tomcat 6.0.29 using more and more RAM until it collapses? I don't think my app is taking all this RAM It is. when I restart it, the RAM usage doesn't go down. Classic symptom of a webapp leaking memory due to holding onto It seems like if Tomcat is leaving garbage in the JVM or something like that. Nope - it's your webapp. Does anybody know what should I do to solve this? Fix your webapp(s). Start with a heap profiler, and look to see what's consuming the space. The problem may also include memory leaks outside of the heap, such as failing to close files, or starting and forgetting about auxiliary threads. Start reading here: http://wiki.apache.org/tomcat/FAQ/Memory http://wiki.apache.org/tomcat/OutOfMemory http://wiki.apache.org/tomcat/MemoryLeakProtection - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Hi Ben, I'm using Apache Commons DBCP (http://commons.apache.org/dbcp/) and I think I'm using it properly. After I perform a SQL sentence, I close the objects (result set, then its statement). I also check what is going on in my DBMS (MySQL), and it shows a normal amount of connections. -Original Message- From: Ben Souther [mailto:b...@souther.us] Sent: Thursday, November 11, 2010 02:06 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? The most common cause of this, that I've seen is the failure to properly close all database connections. If you're using the container managed connection pooling, it is possible that restart your app won't free the ram consumed by any orphaned connections. Without seeing everthing you're doing everything else would be a guess. -Ben On Thu, 2010-11-11 at 13:54 -0500, Brian wrote: Hi, In my Linux machine, I'm using the JVM version 1.6.0_11-b03. On top of that, I only run Tomcat 6.0.29. On that Tomcat installation, I'm running a couple of sites, both of them use exactly the same code, actually it is the same WAR. So they are two apps, but we could consider them as one. The problem is that the RAM usage in the server starts to grow day by day, until Tomcat stops (freezes? hungs?) and my two sites stop working. According to the OS (Linux), it is the Tomcat process that is taking up all that amount of RAM. When I restart Tomcat, it starts fine again, but starts to grow in memory usage day by day again, until it crashes. I don't think my app is taking all this RAM, because when I restart it, the RAM usage doesn't go down. It does only if I restart Tomcat itself, instead of my app running there. It seems like if Tomcat is leaving garbage in the JVM or something like that. According to the Tomcat manager application, in the server status page, the JVM total memory value grows all the time. In this very moment, for example, it says this: Free memory: 178.94 MB Total memory: 565.58 MB Max memory: 692.25 MB. The total memory value is the one that starts growing when Tomcat starts. Does anybody know what should I do to solve this? Thanks! - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? After I perform a SQL sentence, I close the objects (result set, then its statement). In a finally block? If not, you'll leave them open when an exception occurs. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Brian, On 11/11/2010 1:54 PM, Brian wrote: I don't think my app is taking all this RAM, because when I restart it, the RAM usage doesn't go down. That doesn't necessarily mean that your webapp isn't using all that heap space: it's very easy for a webapp to do something foolish and cause all of it's used memory to stay active even after a webapp restart. I disagree with Ben's comment about database connections: there are faster ways to bust your heap than leaking database connections. If you /are/ using Tomcat's container-managed DataSource (which I highly recommend), you should enable all of the abandoned resource tracking features to help fix any resource leaks that you may have there. I agree with Chuck's assessment, but there may be some questions we can ask to help lead you down the right path when using tools such as memory profilers. 0. What kind of OOME are you suffering? Please post any messages that you get when your heap busts. Specifically, is this a regular heap exhaustion or a PermGen exhaustion? 1. Are you seeing any messages in catalina.out when you undeploy of the form your webapp likely has a memory leak? Tomcat 6.0.x has some nice checks on undeploy to see if your webapp is leaking some specific things like threads, etc. 2. Are you re-starting your webapp a lot while Tomcat remains running? Failure to perform some clean-up during webapp unload can cause your entire set of classes loaded in the old webapp to stay in the PermGen space forever. 3. Are you using any caching mechanism in your webapp? Perhaps it needs to be tuned. You should probably check out what's in your application scope: you may have things you didn't expect. 4. Are you using HttpSession for anything bulky? It's possible that you are leaking memory with lots of sessions. When a webapp is stopped, though, all session should be purged and if you have session persistence, they should all come back and take up the same amount of memory. This is a long-shot, but low-hanging fruit. You can get a lot of mileage out of a tool like jmap which comes with the JDK (and JRE?): you can get a heap histogram and look at things that are taking up a lot of space. If you find that you have several tens of megabytes of foo.bar.Baz classes, then you can look at your app to see where you heavy uses of those classes are. To understand #2 a bit better, see Mark's presentation from this year's ApacheCon NA: http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf It's worth a read to understand how simple things like using a logging library can cause PermGen exhaustion after several webapp redeploy operations. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzcRjgACgkQ9CaO5/Lv0PCslgCgs6mzKt5iVkEcid2cel0G5YCn IFgAn23tJppaq7lbmve5ce2laL2exkV3 =+KOk -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Brian, On 11/11/2010 2:26 PM, Brian wrote: I'm using it properly. After I perform a SQL sentence, I close the objects (result set, then its statement). http://blog.christopherschultz.net/index.php/2009/03/16/properly-handling-pooled-jdbc-connections/ I also check what is going on in my DBMS (MySQL), and it shows a normal amount of connections. Good. That's unlikely to be the problem, then. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzcRmsACgkQ9CaO5/Lv0PDJ6wCcCwRu27ibHOaW3XIo2opqvh2c 48kAn0fIecvTDFFSl1MOQU5GxxPatVoM =8Hgg -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Hi Chuck,
Yes, in a Finally block. This is what I do:
void closeRsStmt(ResultSet resultSet)
{
//Close the connection, so it goes back to the pool
Statement stmt=null;
Connection conn=null;
try
{
if (resultSet!=null )
{
stmt = resultSet.getStatement();
resultSet.close();
}
}
catch (SQLException ex)
{
}
try
{
if (stmt!=null )
{
conn=stmt.getConnection();
stmt.close();
}
}
catch (SQLException ex)
{
}
try
{
if (conn!=null)
{
conn.close();
}
}
catch (SQLException ex)
{
}
resultSet = null;
stmt = null;
conn = null;
}
-Original Message-
From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com]
Sent: Thursday, November 11, 2010 02:32 PM
To: Tomcat Users List
Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses?
From: Brian [mailto:bbprefix-m...@yahoo.com]
Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses?
After I perform a SQL sentence, I close the objects (result set, then
its statement).
In a finally block? If not, you'll leave them open when an exception
occurs.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE
PROPRIETARY MATERIAL and is thus for use only by the intended recipient.
If
you received this in error, please contact the sender and delete the
e-mail and its
attachments from all computers.
-
To unsubscribe, e-mail: mailto:users-unsubscr...@tomcat.apache.org
users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: mailto:users-h...@tomcat.apache.org
users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? Yes, in a Finally block. This is what I do: I presume you mean you call your closeRsStmt() method in a finally block, since there certainly aren't any finally clauses in what you published. Your mechanism is a bit suspect, since closing any outer components depends on the retrieval of the outer from an inner. If that fails for any reason, you've left things active. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
-Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Thursday, November 11, 2010 02:39 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Brian, On 11/11/2010 2:26 PM, Brian wrote: I'm using it properly. After I perform a SQL sentence, I close the objects (result set, then its statement). http://blog.christopherschultz.net/index.php/2009/03/16/properly-handling- pooled-jdbc-connections/ I will read this link, thanks!! I also check what is going on in my DBMS (MySQL), and it shows a normal amount of connections. Good. That's unlikely to be the problem, then. That is what I think too. I see, on average, about 30-40 connections in MySQL using its Workbench tool. And that looks fine, that corresponds to the way I configured the pool. But I must say that I hadn't always used the pool. In the past I used to use just one connection, and in many methods I had forgot to close the resultset. You can imagine how much my app crashed! I couldn't even sleep at night! Then I fixed my app so it closes every result set it created, then the statement, and then close the connection so it goes back to the pool. Since that day, my app stopped crashing that often. Now it still crashes, but less often. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 11/11/2010 19:55, Caldarale, Charles R wrote: From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? Yes, in a Finally block. This is what I do: I presume you mean you call your closeRsStmt() method in a finally block, since there certainly aren't any finally clauses in what you published. Your mechanism is a bit suspect +1 Erm, yes. This would be better: http://marc.info/?l=tomcat-userm=128937911023178w=2 p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 11/11/2010 18:54, Brian wrote: I don't think my app is taking all this RAM, because when I restart it, the RAM usage doesn't go down. It does only if I restart Tomcat itself, instead of my app running there. Yes, this is a classic sign of a problem with the app. Reboot Tomcat, restart your app a couple of times (this bit is important). Connect to the Tomcat instance using JConsole, navigate the MBeans, to Catalina Hosts (your hostname), then select the Operations tab, under which you'll see a button called findReloadContextMemoryLeaks. Push the button. It will return a list of app names if Tomcat can detect ones with memory leaks. NB No results doesn't necessarily mean your app isn't leaking. p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
-Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Thursday, November 11, 2010 02:55 PM To: Tomcat Users List Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? Yes, in a Finally block. This is what I do: I presume you mean you call your closeRsStmt() method in a finally block, since there certainly aren't any finally clauses in what you published. Oh, yes, I forgot to clarify that. I call that method (that I showed in my last email) from the finally block that is present after all my pieces of code that use SQL. Your mechanism is a bit suspect, since closing any outer components depends on the retrieval of the outer from an inner. If that fails for any reason, you've left things active. You know what? You are right! I will check my logic again. Maybe I should start putting some messages in the log, inside the catch blocks that are empty. Maybe you are right, and those pieces of code are failing, but my catch block doesn't do anything about it, not even leaving something in the log. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Thanks for the link. I will read it asap. -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Thursday, November 11, 2010 03:01 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 11/11/2010 19:55, Caldarale, Charles R wrote: From: Brian [mailto:bbprefix-m...@yahoo.com] Subject: RE: Tomcat 6.0.29 using more and more RAM until it collapses? Yes, in a Finally block. This is what I do: I presume you mean you call your closeRsStmt() method in a finally block, since there certainly aren't any finally clauses in what you published. Your mechanism is a bit suspect +1 Erm, yes. This would be better: http://marc.info/?l=tomcat-userm=128937911023178w=2 p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
It seems that it is my app the source of the problem. I guess I was in denial. :-( I haven't ever heard of Jconsole before, but I will install it ASAP and do what you adviced me. Thanks a lot! -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Thursday, November 11, 2010 03:06 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 11/11/2010 18:54, Brian wrote: I don't think my app is taking all this RAM, because when I restart it, the RAM usage doesn't go down. It does only if I restart Tomcat itself, instead of my app running there. Yes, this is a classic sign of a problem with the app. Reboot Tomcat, restart your app a couple of times (this bit is important). Connect to the Tomcat instance using JConsole, navigate the MBeans, to Catalina Hosts (your hostname), then select the Operations tab, under which you'll see a button called findReloadContextMemoryLeaks. Push the button. It will return a list of app names if Tomcat can detect ones with memory leaks. NB No results doesn't necessarily mean your app isn't leaking. p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.29 using more and more RAM until it collapses?
On 11/11/2010 20:14, Brian wrote: It seems that it is my app the source of the problem. I guess I was in denial. :-( I haven't ever heard of Jconsole before, but I will install it ASAP and do what you adviced me. Thanks a lot! If you have a JDK installed, you already have it. It's in the java/bin directory. p -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Thursday, November 11, 2010 03:06 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 11/11/2010 18:54, Brian wrote: I don't think my app is taking all this RAM, because when I restart it, the RAM usage doesn't go down. It does only if I restart Tomcat itself, instead of my app running there. Yes, this is a classic sign of a problem with the app. Reboot Tomcat, restart your app a couple of times (this bit is important). Connect to the Tomcat instance using JConsole, navigate the MBeans, to Catalina Hosts (your hostname), then select the Operations tab, under which you'll see a button called findReloadContextMemoryLeaks. Push the button. It will return a list of app names if Tomcat can detect ones with memory leaks. NB No results doesn't necessarily mean your app isn't leaking. p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
RE: Tomcat 6.0.29 using more and more RAM until it collapses?
Hi Pid, I did it, but shows no results. Anyway, it was nice to learn about Jconsole. Now I wonder what is the tool I could use to inspect the objets inside my app, and see which ones are using all the memory. -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Thursday, November 11, 2010 03:06 PM To: Tomcat Users List Subject: Re: Tomcat 6.0.29 using more and more RAM until it collapses? On 11/11/2010 18:54, Brian wrote: I don't think my app is taking all this RAM, because when I restart it, the RAM usage doesn't go down. It does only if I restart Tomcat itself, instead of my app running there. Yes, this is a classic sign of a problem with the app. Reboot Tomcat, restart your app a couple of times (this bit is important). Connect to the Tomcat instance using JConsole, navigate the MBeans, to Catalina Hosts (your hostname), then select the Operations tab, under which you'll see a button called findReloadContextMemoryLeaks. Push the button. It will return a list of app names if Tomcat can detect ones with memory leaks. NB No results doesn't necessarily mean your app isn't leaking. p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
