Re: Suggestions for really getting to the bottom of memory leak issues on hot-deploys
[from your blog entry] I also think that logging leaks may be more helpful than using part of manager, because many don't deploy manager (they remove it). it is logged when the application is stopped, on recent tomcat 6 and 7. I think your approach to finding leaks by performing comparisons is not the most efficient. What I usually do to find the leak is to use MAT (eclipse memory analyzer, as you used too) and its class loader explorer. There I find the instance of org.apache.catalina.loader.WebappClassLoader that corresponds to my webapp and right-click to ask path to GC roots/exclude all phantom/weak/soft, etc. references. There I usually see the cause of the leak vey quickly. I you find no strong reference to your classloaders then you might have encountered some JDK bug... You should use a recent one to avoid some (at least 6.0u21). Sylvain On 10 mai 2011, at 21:55, Gary Weaver wrote: Here's what I did and what little I came up with: http://stufftohelpyouout.blogspot.com/2011/05/diagnosing-webappportlet-hot-deploy.html I'm definitely not an expert at diagnosing leaks, so if you have any recommendations/comments, please let me know. I unfortunately started off just thinking I could tune JVM settings, not just by bumping up permgen but by adding things like: -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled But, I was (really) wrong as you can see in the post, along with the various tools used to try to identify what was causing the issue(s). It doesn't seem like determining exactly what is leaking memory (at least in the case of permgen and hot-redeploys/hot-deploys) is a whole lot easier than it was years ago, unless I'm doing something wrong, which I most likely am. Thanks in advance for any feedback! Gary - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Suggestions for really getting to the bottom of memory leak issues on hot-deploys
On 5/10/11 8:55 PM, Gary Weaver wrote: Here's what I did and what little I came up with: http://stufftohelpyouout.blogspot.com/2011/05/diagnosing-webappportlet-hot-deploy.html I'm definitely not an expert at diagnosing leaks, so if you have any recommendations/comments, please let me know. I unfortunately started off just thinking I could tune JVM settings, not just by bumping up permgen but by adding things like: -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled But, I was (really) wrong as you can see in the post, along with the various tools used to try to identify what was causing the issue(s). It doesn't seem like determining exactly what is leaking memory (at least in the case of permgen and hot-redeploys/hot-deploys) is a whole lot easier than it was years ago, unless I'm doing something wrong, which I most likely am. Thanks in advance for any feedback! Gary Did you examine the Tomcat 7 logs during redeploy cycles? VisualVM can also take and/or open a heap dump. p signature.asc Description: OpenPGP digital signature
Re: Suggestions for really getting to the bottom of memory leak issues on hot-deploys
binLmaU7goP1J.bin Description: PGP/MIME version identification
Re: Suggestions for really getting to the bottom of memory leak issues on hot-deploys
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Gary, On 5/10/2011 4:33 PM, Gary Weaver wrote: Chris, On Tue, May 10, 2011 at 4:06 PM, Christopher Schultz ch...@christopherschultz.net wrote: http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf Thanks! That is a great presentation! It really is. Having a decent memory profiler can really help things out. If you can, try upgrading to the latest version of either Tomcat 6 or Tomcat 7... there are some speculative memory leak prevention techniques that they use to try to eliminate some memory leaks that webapps cannot hope to eliminate by themselves -- things like JVM-provided classes that store references to the context class loader. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk3KmBQACgkQ9CaO5/Lv0PBRdQCeMAdx6jA+RasVei8e1cprTXoy mWEAoIlg5+VY+ZXQptXkdVsh8OV8gGdo =8hjI -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Suggestions for really getting to the bottom of memory leak issues on hot-deploys
Here's what I did and what little I came up with: http://stufftohelpyouout.blogspot.com/2011/05/diagnosing-webappportlet-hot-deploy.html I'm definitely not an expert at diagnosing leaks, so if you have any recommendations/comments, please let me know. I unfortunately started off just thinking I could tune JVM settings, not just by bumping up permgen but by adding things like: -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled But, I was (really) wrong as you can see in the post, along with the various tools used to try to identify what was causing the issue(s). It doesn't seem like determining exactly what is leaking memory (at least in the case of permgen and hot-redeploys/hot-deploys) is a whole lot easier than it was years ago, unless I'm doing something wrong, which I most likely am. Thanks in advance for any feedback! Gary
Re: Suggestions for really getting to the bottom of memory leak issues on hot-deploys
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Gary, On 5/10/2011 3:55 PM, Gary Weaver wrote: http://stufftohelpyouout.blogspot.com/2011/05/diagnosing-webappportlet-hot-deploy.html Generally speaking, when you have a permgen leak across webapp redeployments, this is the situation: 1. Webapp invokes some code that stores a reference to the webapp's WebappClassLoader 2. That reference is not cleared during undeploy [this is the leak] The big problem is that the WebappClassLoader keeps a reference to all java.lang.Class objects it loads, so it can be quite bulky when it leaks, depending on exactly how many classes your webapp has locally. markt has a great presentation which lays out the anatomy of these types of memory leaks, how to diagnose them, and often how to fix them. These are slides from a presentation at last year's ApacheCon NA. It's /very/ readable. http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk3JmrQACgkQ9CaO5/Lv0PAwWwCfQNuuTZnw9JYqgVSdQ/daVABG 6jQAnRb1VJScgOjH8J73EKaWZLukKcqf =Bvw2 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Suggestions for really getting to the bottom of memory leak issues on hot-deploys
Chris, On Tue, May 10, 2011 at 4:06 PM, Christopher Schultz ch...@christopherschultz.net wrote: http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf Thanks! That is a great presentation! Gary
