Re: [Resin-interest] Out of PermGen space
On 05/29/2012 09:51 PM, Mattias Jiderhamn wrote: On 05/29/2012 12:27 PM, Mattias Jiderhamn wrote: - Original Message - Subject: Re: [Resin-interest] Out of PermGen space Date: Fri, 27 Apr 2012 11:23:41 +0200 From: Mattias Jiderhamnmj-li...@expertsystems.se I should also mention, that I have an open bug report on Resin 4.0.27 which seems to cause classloader leaks. http://bugs.caucho.com/view.php?id=5010 Has anyone had time to confirm this bug? It seems to be triggered more often when using JSF, so when developing JSF applications I need to restart the server over and over (no difference whether HotSwap is enabled or not). Would be happy to know if there is hope that this will be fixed. Do you have a trace to root on one of those? (We've finally found the bugtrack upload corruption issue, but haven't updated that server yet, so the current pngs aren't readable.) See http://jiderhamn.se/resin4027.png + http://jiderhamn.se/resin4027b.png Let me know if I can provide any additional info. Thanks. The first one is straightforward to fix (the AnnotationLiteral is a surprising dependency, though.) I'm not sure how the second one is possible (the one where your application's bean is in the ResinSystem CDI context.) I assume that bean is just scanned normally? -- Scott /Mattias ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Out of PermGen space
- Original Message - Subject: Re: [Resin-interest] Out of PermGen space Date: Fri, 27 Apr 2012 11:23:41 +0200 From: Mattias Jiderhamn mj-li...@expertsystems.se I should also mention, that I have an open bug report on Resin 4.0.27 which seems to cause classloader leaks. http://bugs.caucho.com/view.php?id=5010 Has anyone had time to confirm this bug? It seems to be triggered more often when using JSF, so when developing JSF applications I need to restart the server over and over (no difference whether HotSwap is enabled or not). Would be happy to know if there is hope that this will be fixed. /Mattias In pretty much every heap dump I analyze for ZombieClassLoaderMarker these days, com.caucho.config.inject.InjectManager turns up somewhere in the path to root. Not sure if it will fix all the problems, but it sure would be interesting to try a Resin version without this issue. P.S. I did some experimenting with Resin Pro vs Open Source today, and on both found some strange paths leading to JNI Local com.caucho.env.thread2.ResinThread2. Disabling JNI (on Windows, removing resin_os.dll) seemed to postpone the crash (which again included InjectManager) substantially. Not enought statistics to draw conclustions, but there *may* be something fishy about the JNI connection under Windows. /Mattias -- /Mattias ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Out of PermGen space
On 05/29/2012 12:27 PM, Mattias Jiderhamn wrote: - Original Message - Subject: Re: [Resin-interest] Out of PermGen space Date: Fri, 27 Apr 2012 11:23:41 +0200 From: Mattias Jiderhamnmj-li...@expertsystems.se I should also mention, that I have an open bug report on Resin 4.0.27 which seems to cause classloader leaks. http://bugs.caucho.com/view.php?id=5010 Has anyone had time to confirm this bug? It seems to be triggered more often when using JSF, so when developing JSF applications I need to restart the server over and over (no difference whether HotSwap is enabled or not). Would be happy to know if there is hope that this will be fixed. Do you have a trace to root on one of those? (We've finally found the bugtrack upload corruption issue, but haven't updated that server yet, so the current pngs aren't readable.) -- Scott /Mattias In pretty much every heap dump I analyze for ZombieClassLoaderMarker these days, com.caucho.config.inject.InjectManager turns up somewhere in the path to root. Not sure if it will fix all the problems, but it sure would be interesting to try a Resin version without this issue. P.S. I did some experimenting with Resin Pro vs Open Source today, and on both found some strange paths leading to JNI Local com.caucho.env.thread2.ResinThread2. Disabling JNI (on Windows, removing resin_os.dll) seemed to postpone the crash (which again included InjectManager) substantially. Not enought statistics to draw conclustions, but there *may* be something fishy about the JNI connection under Windows. /Mattias -- /Mattias ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Out of PermGen space
On 05/29/2012 12:27 PM, Mattias Jiderhamn wrote: - Original Message - Subject: Re: [Resin-interest] Out of PermGen space Date: Fri, 27 Apr 2012 11:23:41 +0200 From: Mattias Jiderhamnmj-li...@expertsystems.se I should also mention, that I have an open bug report on Resin 4.0.27 which seems to cause classloader leaks. http://bugs.caucho.com/view.php?id=5010 Has anyone had time to confirm this bug? It seems to be triggered more often when using JSF, so when developing JSF applications I need to restart the server over and over (no difference whether HotSwap is enabled or not). Would be happy to know if there is hope that this will be fixed. Do you have a trace to root on one of those? (We've finally found the bugtrack upload corruption issue, but haven't updated that server yet, so the current pngs aren't readable.) See http://jiderhamn.se/resin4027.png + http://jiderhamn.se/resin4027b.png Let me know if I can provide any additional info. /Mattias ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Out of PermGen space
If it is not a leak (classes are being cleaned up upon webapp reload but a large number is being created), simply increase max perm size. Bill On Wed, Apr 25, 2012 at 11:47 PM, Matt White mwh...@leporidae.net wrote: On 4/24/2012 5:13 PM, Bill Au wrote: Wow, something I'm actually qualified to talk about on this list! :) Out of PermGen space is almost always caused by a classloader leak which occurs when a webapp is reloaded. It could be caused by either your own code, third-party code, or in some case Java core classes. I debug PermGen errors quite a bit. Some of our apps have a cache of dynamically generated classes (Drools does this) that, if held onto long enough, will make the the JVM run out of PermGen. This isn't really a mistake, it's just how it works, sadly. You need to take heap dumps before and after webapp reload and use a heap analyzer to see what is holding onto the leaked classloader(s). That's how I debug them. Take a series of heap dumps and start comparing the differences between them -- YourKit even makes this easy for you with a tool that diffs heap dumps. - Matt ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Out of PermGen space
On 04/25/2012 08:47 PM, Matt White wrote: On 4/24/2012 5:13 PM, Bill Au wrote: Wow, something I'm actually qualified to talk about on this list! :) Out of PermGen space is almost always caused by a classloader leak which occurs when a webapp is reloaded. It could be caused by either your own code, third-party code, or in some case Java core classes. I debug PermGen errors quite a bit. Some of our apps have a cache of dynamically generated classes (Drools does this) that, if held onto long enough, will make the the JVM run out of PermGen. This isn't really a mistake, it's just how it works, sadly. You need to take heap dumps before and after webapp reload and use a heap analyzer to see what is holding onto the leaked classloader(s). That's how I debug them. Take a series of heap dumps and start comparing the differences between them -- YourKit even makes this easy for you with a tool that diffs heap dumps. If you're using Resin 4.0, have you tried looking for the ZombieClassLoaderMarker? It's designed to make that tracing easier. -- Scott - Matt ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Out of PermGen space
Thanks, Mattias, that's cool! -- Rick On Apr 24, 2012, at 22:17 , Mattias Jiderhamn wrote: Hi Rick. After having had these issues for years, I started blogging about it and how to find classloader leaks [1]. I also compiled a list of API calls and third party libraries known to trigger these leaks [2], and as you can see, it is quite common both to cause these problems yourself and to have them caused by some dependency. Having done the research, I ended up creating a small library that you can add to you web application, that will clean up strong references keeping your classloader from being garbage collected [3]. Somewhat like Tomcat has built in, but taking care of more of the known problems. I suggest you add this library to you app and watch the logs (feel free to report your findings). Having said that, I should admit that I'm still seeing PermGen crashes running under Resin, even when there are no visible strong references to my classloader. I have not had time to investigate whether this is caused by Resin (such as the HotSwap capability), or if it is some JVM bug (and possibly something with IntelliJ, as the problem increases using the IntelliJ Resin plugin). I should probably try turning HotSwap off, or try another application server - or another JVM. Someday... Good luck! /Mattias 1: http://java.jiderhamn.se/2011/12/11/classloader-leaks-i-how-to-find-classloader-leaks-with-eclipse-memory-analyser-mat/ 2: http://java.jiderhamn.se/2012/02/26/classloader-leaks-v-common-mistakes-and-known-offenders/ 3: http://java.jiderhamn.se/2012/03/04/classloader-leaks-vi-this-means-war-leak-prevention-library/ - Original Message - Subject: [Resin-interest] Out of PermGen space Date: Tue, 24 Apr 2012 13:54:40 -0700 From: Rick Mann rm...@latencyzero.com When I'm making changes to the code of a webapp, Resin kindly reloads it for me. I can usually get a handful of reloads in before Resin complains about being out of PermGen space. Is there something I'm doing wrong in my app that it leaks like this? -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest -- /Mattias ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Out of PermGen space
On 4/24/2012 5:13 PM, Bill Au wrote: Wow, something I'm actually qualified to talk about on this list! :) Out of PermGen space is almost always caused by a classloader leak which occurs when a webapp is reloaded. It could be caused by either your own code, third-party code, or in some case Java core classes. I debug PermGen errors quite a bit. Some of our apps have a cache of dynamically generated classes (Drools does this) that, if held onto long enough, will make the the JVM run out of PermGen. This isn't really a mistake, it's just how it works, sadly. You need to take heap dumps before and after webapp reload and use a heap analyzer to see what is holding onto the leaked classloader(s). That's how I debug them. Take a series of heap dumps and start comparing the differences between them -- YourKit even makes this easy for you with a tool that diffs heap dumps. - Matt ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
[Resin-interest] Out of PermGen space
When I'm making changes to the code of a webapp, Resin kindly reloads it for me. I can usually get a handful of reloads in before Resin complains about being out of PermGen space. Is there something I'm doing wrong in my app that it leaks like this? -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Out of PermGen space
On 24.04.2012 22:54, Rick Mann wrote: When I'm making changes to the code of a webapp, Resin kindly reloads it for me. I can usually get a handful of reloads in before Resin complains about being out of PermGen space. Is there something I'm doing wrong in my app that it leaks like this? Nope. The standard setting is pretty low. I usually set MaxPermSize to 256M. ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Out of PermGen space
On Apr 24, 2012, at 14:32 , Olaf Krische wrote: On 24.04.2012 22:54, Rick Mann wrote: When I'm making changes to the code of a webapp, Resin kindly reloads it for me. I can usually get a handful of reloads in before Resin complains about being out of PermGen space. Is there something I'm doing wrong in my app that it leaks like this? Nope. The standard setting is pretty low. I usually set MaxPermSize to 256M. Mine is set higher. It runs fine for some time. Eventually, it'll try to reload the app, and get this error. Memory in my app (or in Resin) is leaking, and I don't know why (nor enough about java debugging to figure out where the leak is). -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Out of PermGen space
Well, yes and no. As I understand the problem, it really got bad around the Java 5 timeframe because of the addition of Enumerations to the language. What Resin does (and all auto-reloading Java containers do) is to create a ClassLoader that contains all the code for your application. When it senses a change to your code, it loads that new code into a brand new ClassLoader and releases the old one to be garbage collected once it's done processing it's active requests. The problem is that, since Enumerations are guaranteed to work with the == operator, even when serialized/deserialized between different computers, Java treats them special, by keeping them in the Permanent Generation (so their internal ID's won't change). Unfortunately, each time the app is loaded a new set of Enumerations takes up more space in the precious PermGen until it finally blows its lid. So, theoretically, you could use less PermGen by limiting Enumeration use, but that's really not a realistic response. Before Java became property of Oracle, there was some talk about fixing this problem at the JVM level, but I haven't heard anything in quite a while. (*Chris*) On Tue, Apr 24, 2012 at 1:54 PM, Rick Mann rm...@latencyzero.com wrote: When I'm making changes to the code of a webapp, Resin kindly reloads it for me. I can usually get a handful of reloads in before Resin complains about being out of PermGen space. Is there something I'm doing wrong in my app that it leaks like this? -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Out of PermGen space
Out of PermGen space is almost always caused by a classloader leak which occurs when a webapp is reloaded. It could be caused by either your own code, third-party code, or in some case Java core classes. You need to take heap dumps before and after webapp reload and use a heap analyzer to see what is holding onto the leaked classloader(s). Bill On Tue, Apr 24, 2012 at 5:41 PM, Chris Pratt thechrispr...@gmail.comwrote: Well, yes and no. As I understand the problem, it really got bad around the Java 5 timeframe because of the addition of Enumerations to the language. What Resin does (and all auto-reloading Java containers do) is to create a ClassLoader that contains all the code for your application. When it senses a change to your code, it loads that new code into a brand new ClassLoader and releases the old one to be garbage collected once it's done processing it's active requests. The problem is that, since Enumerations are guaranteed to work with the == operator, even when serialized/deserialized between different computers, Java treats them special, by keeping them in the Permanent Generation (so their internal ID's won't change). Unfortunately, each time the app is loaded a new set of Enumerations takes up more space in the precious PermGen until it finally blows its lid. So, theoretically, you could use less PermGen by limiting Enumeration use, but that's really not a realistic response. Before Java became property of Oracle, there was some talk about fixing this problem at the JVM level, but I haven't heard anything in quite a while. (*Chris*) On Tue, Apr 24, 2012 at 1:54 PM, Rick Mann rm...@latencyzero.com wrote: When I'm making changes to the code of a webapp, Resin kindly reloads it for me. I can usually get a handful of reloads in before Resin complains about being out of PermGen space. Is there something I'm doing wrong in my app that it leaks like this? -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Out of PermGen space
Hello Chris, this actually makes me wonder: are enums eating so much space, especially when compared to those things, that you release with the classloader? Heard about this for the first time. Interesting. On 24.04.2012 23:41, Chris Pratt wrote: that, since Enumerations are guaranteed to work with the == operator, even when serialized/deserialized between different computers, Java treats them special, by keeping them in the Permanent Generation (so their internal ID's won't change). Unfortunately, each time the app is loaded a new set of Enumerations takes up more space in the precious PermGen until it finally blows its lid. So, theoretically, you could use less PermGen by limiting Enumeration use, but that's really not a realisticOn ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Out of PermGen space
On 04/24/2012 03:13 PM, Bill Au wrote: Out of PermGen space is almost always caused by a classloader leak which occurs when a webapp is reloaded. It could be caused by either your own code, third-party code, or in some case Java core classes. You need to take heap dumps before and after webapp reload and use a heap analyzer to see what is holding onto the leaked classloader(s). As a quick check, if you're using Resin 4.0 Pro, the PDF dump shows ZombieClassLoaderMarker in the heap dump section. Or you can use a heap analyzer and look for ZombieClassLoaderMarker and do a search-to-root on it. In Resin, a zombie classloader is one that Resin expects to be garbage collected. -- Scott Bill On Tue, Apr 24, 2012 at 5:41 PM, Chris Pratt thechrispr...@gmail.com mailto:thechrispr...@gmail.com wrote: Well, yes and no. As I understand the problem, it really got bad around the Java 5 timeframe because of the addition of Enumerations to the language. What Resin does (and all auto-reloading Java containers do) is to create a ClassLoader that contains all the code for your application. When it senses a change to your code, it loads that new code into a brand new ClassLoader and releases the old one to be garbage collected once it's done processing it's active requests. The problem is that, since Enumerations are guaranteed to work with the == operator, even when serialized/deserialized between different computers, Java treats them special, by keeping them in the Permanent Generation (so their internal ID's won't change). Unfortunately, each time the app is loaded a new set of Enumerations takes up more space in the precious PermGen until it finally blows its lid. So, theoretically, you could use less PermGen by limiting Enumeration use, but that's really not a realistic response. Before Java became property of Oracle, there was some talk about fixing this problem at the JVM level, but I haven't heard anything in quite a while. (*Chris*) On Tue, Apr 24, 2012 at 1:54 PM, Rick Mann rm...@latencyzero.com mailto:rm...@latencyzero.com wrote: When I'm making changes to the code of a webapp, Resin kindly reloads it for me. I can usually get a handful of reloads in before Resin complains about being out of PermGen space. Is there something I'm doing wrong in my app that it leaks like this? -- Rick ___ resin-interest mailing list resin-interest@caucho.com mailto:resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com mailto:resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
[Resin-interest] Out of PermGen space
Hi Rick. After having had these issues for years, I started blogging about it and how to find classloader leaks [1]. I also compiled a list of API calls and third party libraries known to trigger these leaks [2], and as you can see, it is quite common both to cause these problems yourself and to have them caused by some dependency. Having done the research, I ended up creating a small library that you can add to you web application, that will clean up strong references keeping your classloader from being garbage collected [3]. Somewhat like Tomcat has built in, but taking care of more of the known problems. I suggest you add this library to you app and watch the logs (feel free to report your findings). Having said that, I should admit that I'm still seeing PermGen crashes running under Resin, even when there are no visible strong references to my classloader. I have not had time to investigate whether this is caused by Resin (such as the HotSwap capability), or if it is some JVM bug (and possibly something with IntelliJ, as the problem increases using the IntelliJ Resin plugin). I should probably try turning HotSwap off, or try another application server - or another JVM. Someday... Good luck! /Mattias 1: http://java.jiderhamn.se/2011/12/11/classloader-leaks-i-how-to-find-classloader-leaks-with-eclipse-memory-analyser-mat/ 2: http://java.jiderhamn.se/2012/02/26/classloader-leaks-v-common-mistakes-and-known-offenders/ 3: http://java.jiderhamn.se/2012/03/04/classloader-leaks-vi-this-means-war-leak-prevention-library/ - Original Message - Subject: [Resin-interest] Out of PermGen space Date: Tue, 24 Apr 2012 13:54:40 -0700 From: Rick Mann rm...@latencyzero.com When I'm making changes to the code of a webapp, Resin kindly reloads it for me. I can usually get a handful of reloads in before Resin complains about being out of PermGen space. Is there something I'm doing wrong in my app that it leaks like this? -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest -- /Mattias ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
