Memory leaking on [un|re]load: WebappClassLoader isn't GC'ed
(Apparently sent this to the wrong address first time)
Hi!
I have a memory-leak.
That is, I can't start my full application and have unload-capability. If I
"comment out" (by adding 'if (false) { ... }') large chuncks of the system, I
manage to get it up, _and_ get it down again!
Currently, pretty much the whole application have to be commented out to enable
webapp unloading to actually let the system be GC'ed. Both if I comment in the
starting of WebMacro, or the "subsystem" that parses the system XML
configuration (which uses on jdom), I can't unload.
The problem seems to be that I get a large cluster with somehow-reacable
objects forming - all having the not-GC'ed WebappClassLoader somewhere in
common.
Is there any documentation out there that can give me a definitive clue? I've
used VERY MUCH time on this problem now, and searched high and low for answers.
I've used JProfiler and YourKit to my limit too, and still can't seem to get an
understanding: Using the "find GC roots", I sometimes get an _elaborate_
diagram including lots of tomcat and sun mbean-stuff that I don't know (!), but
often there is just an _annoying_ "other gc root" or "Other GC".
Any help appreciated.
Regards,
Endre
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory leaking on [un|re]load: WebappClassLoader isn't GC'ed
Good day, I have encountered a similar problem while deploying webapps. In my case I noticed that after the sixth deployment I would encounter OutOfMemoryError when I tried to use the application. After trawling the archives and running the tomcat jvm with the -XX:+PrintGCDetails option I discovered that I run out of PermSpace. Reading this posting http://marc.theaimsgroup.com/?l=tomcat-user&m=113532811904238&w=2 from Rodrigo Ruiz I discover that the memory leak is caused by static classes (aka Singletons) that are dotted all over support jars in my classpath. For me to solve my problem I would need to go on one hell of an anti-singleton crusade, and tempting as it, is I haven't the time. Perhaps this is the cause of your problems as well? Kind regards, Mike Fowler Registered Linux user: 379787 "I could be a genius if I just put my mind to it, and I, I could do anything, if only I could get 'round to it" -PULP 'Glory Days' - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory leaking on [un|re]load: WebappClassLoader isn't GC'ed
Il giorno mer, 25/01/2006 alle 15.34 +, Mike Fowler ha scritto: > Good day, > > I have encountered a similar problem while deploying webapps. In my case > I noticed that after the sixth deployment I would encounter > OutOfMemoryError when I tried to use the application. After trawling the > archives and running the tomcat jvm with the -XX:+PrintGCDetails option > I discovered that I run out of PermSpace. Reading this posting > http://marc.theaimsgroup.com/?l=tomcat-user&m=113532811904238&w=2 from > Rodrigo Ruiz I discover that the memory leak is caused by static classes > (aka Singletons) that are dotted all over support jars in my classpath. > For me to solve my problem I would need to go on one hell of an > anti-singleton crusade, and tempting as it, is I haven't the time. > Perhaps this is the cause of your problems as well? > > Kind regards, > It's a known problem caused by a lot of different situations. It's sad that a lot of important applications suffer of this problem. I'm working with Axis, and just trying to reload the bundled webapp never release the old WebappClassloader. I think this is serious: hot deployment becomes totally useless and you have to restart tomcat each time you deploy a new version :-( Bye, Davide Romanini - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory leaking on [un|re]load: WebappClassLoader isn't GC'ed
Davide Romanini wrote: I think this is serious: hot deployment becomes totally useless and you have to restart tomcat each time you deploy a new version :-( I agree, and from what I understand of the problem there is not a whole lot that can be done at the Tomcat level, it appears to be down to the JVM itself. For example, a collegue has experienced a similar problem in WebLogic. I run Tomcat 5.0.28 in Sun's HotSpot Client VM 1.4.2_09. I wonder if other JVM implementations with presumably different GC implementations suffer this problem as well? I've always hated the Singleton pattern and it's overuse. To me it's use seems like a bad OO design. I do not use the singleton in my designs, but I've been stung by "singleton hell" because of, IMHO, poorly designed support libraries that rely far to heavily on the use of singletons. (I feel a little less frustrated with this problem now!). Kind regards, Mike Fowler Registered Linux user: 379787 "I could be a genius if I just put my mind to it, and I, I could do anything, if only I could get 'round to it" -PULP 'Glory Days' - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory leaking on [un|re]load: WebappClassLoader isn't GC'ed
--- Davide Romanini <[EMAIL PROTECTED]> wrote: > Il giorno mer, 25/01/2006 alle 15.34 +, Mike > Fowler ha scritto: > > Good day, > > > > I have encountered a similar problem while > deploying webapps. In my case > > I noticed that after the sixth deployment I would > encounter > > OutOfMemoryError when I tried to use the > application. After trawling the > > archives and running the tomcat jvm with the > -XX:+PrintGCDetails option > > I discovered that I run out of PermSpace. Reading > this posting > > > http://marc.theaimsgroup.com/?l=tomcat-user&m=113532811904238&w=2 > from > > Rodrigo Ruiz I discover that the memory leak is > caused by static classes > > (aka Singletons) that are dotted all over support > jars in my classpath. > > For me to solve my problem I would need to go on > one hell of an > > anti-singleton crusade, and tempting as it, is I > haven't the time. > > Perhaps this is the cause of your problems as > well? > > > > Kind regards, > > > > It's a known problem caused by a lot of different > situations. It's sad > that a lot of important applications suffer of this > problem. I'm working > with Axis, and just trying to reload the bundled > webapp never release > the old WebappClassloader. > > I think this is serious: hot deployment becomes > totally useless and you > have to restart tomcat each time you deploy a new > version :-( > > Bye, > Davide Romanini Don't use shared libraries and you shouldn't have this problem. If all of the statics are in your WEB-INF directory then you won't have an issue with unloading classes and class loaders. Wade - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory leaking on [un|re]load: WebappClassLoader isn't GC'ed
Il giorno mer, 25/01/2006 alle 11.12 -0800, Wade Chandler ha scritto: > Don't use shared libraries and you shouldn't have this > problem. If all of the statics are in your WEB-INF > directory then you won't have an issue with unloading > classes and class loaders. > It would be great if could be so simple. But it is not, trust me, my shared folder is empty, and the common/lib folder contains only jdbc drivers and javamail implementation (needed by Tomcat if you want to use JNDI mail sessions). All the libraries of the webapp are in WEB-INF/lib folder. As I said in previous mail, I tried to reload the distribution version of Axis (just as is, without other libs) and it causes the problem. It seems that something occuring during AxisServlet startup (it looks for his configuration using a complex discovery process) causes some class of a parent classloader to hold a reference to WebappClassloader so it cannot be GC'ed when it's reloaded. Surely it's not a Tomcat problem anyway... Bye, Davide Romanini - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory leaking on [un|re]load: WebappClassLoader isn't GC'ed
| > It's a known problem caused by a lot of different | > situations. It's sad | > that a lot of important applications suffer of this | > problem. I'm working | > with Axis, and just trying to reload the bundled | > webapp never release | > the old WebappClassloader. | > | > I think this is serious: hot deployment becomes | > totally useless and you | > have to restart tomcat each time you deploy a new | > version :-( | > | > Bye, | > Davide Romanini | | Don't use shared libraries and you shouldn't have this | problem. If all of the statics are in your WEB-INF | directory then you won't have an issue with unloading | classes and class loaders. That was a thought that popped into my head too at some point: it might be XML parsing? That it loads Tomcat's instance of xerces, and that this again ends up holding some reference to some piece of my data structures, which again holds the class, which again holds the WebappClassLoader? Or some stuff like that? - But I use Java 1.5? Btw, vote for this one: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6254531 (Read what the guy _actually_ points out, and then ignore the Sun-engineer's evaluation, and read the reply.) On a side note: It is annoying that JProfiler assumes that a static field is a "GC Root". YourKit is way better on this particular aspect, but I like the rest of JProfiler better! But it is even more annoying that the bleedin' SUN JVM and the JVMTI API even has a concept of "Other GC Root" - the guy that invented that JVMTI fieldname should have stopped for a moment and thought about "What can _possibly_ constitute "other" GC roots? I must be a sloppy coder, I should quit this job: Inventing a special type for my non-understanding of garbage collection is a clear sign of incompetence". And what about the guy that implemented the JVMTI interface in SUN's JVM: "ah lookey here, an "I can't bother to find the source of this root"-type root! I'll use that for pretty much anything that's in here, coz' I'm so fantastically lazy!". The problem is that most of the time, the WebappClassLoader that apparently is holding my entire universe and just won't let go, has "Other GC Root" as its GC Root. Now, THAT'S really helpful, thanks a lot!! Well, rather fed up with leaking java, Endre. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory leaking on [un|re]load: WebappClassLoader isn't GC'ed
On Thu, 26 Jan 2006, Davide Romanini wrote: | Il giorno mer, 25/01/2006 alle 11.12 -0800, Wade Chandler ha scritto: | | > Don't use shared libraries and you shouldn't have this | > problem. If all of the statics are in your WEB-INF | > directory then you won't have an issue with unloading | > classes and class loaders. | > | | It would be great if could be so simple. But it is not, trust me, my | shared folder is empty, and the common/lib folder contains only jdbc | drivers and javamail implementation (needed by Tomcat if you want to use | JNDI mail sessions). All the libraries of the webapp are in WEB-INF/lib | folder. As I said in previous mail, I tried to reload the distribution | version of Axis (just as is, without other libs) and it causes the | problem. It seems that something occuring during AxisServlet startup (it | looks for his configuration using a complex discovery process) causes | some class of a parent classloader to hold a reference to | WebappClassloader so it cannot be GC'ed when it's reloaded. But, try to reload the tomcat-docs webapp. That happens without problems. And as I mentioned, if I comment out pretty much the entire "startup" of my webapp, it can be reloaded just fine. In particular, I definately can't start WebMacro (which (tries to) read files through both the classloader and File and URL), nor read the Configuration (which uses JDOM, which uses SAX) and still have reload-functionality. This _really_ bugs me! | | Surely it's not a Tomcat problem anyway... That I really hope, but as of now I can't rule it out. I've tried with several Tomcat versions (4, 5, 5.5), all with the same result. But I'll give Jetty a try too..! However, all the objects that are lingering have some root going through some mbean-server and whatnot, and ends up in static fields of tomcat. Unless these instances also are held by "my" WebappClassLoader (which due to the names of the classes I tend to not believe; they seem so "rooty"), there is _something_ that in certain situations keeps some reference to some object of the webapp. But I'm not at the bottom of those objects yet. Remember that only _one single reference_ to _any_ object of the webapp will hold pretty much the _entire_ webapp from unloading: This is because each object holds a ref to its class, which again holds a ref to its classloader, which again holds refs to all classes loaded in the classloader, which again have static fields, which again probably holds a whole bunch of your actual objects. ThreadLocals as they are implemented in Java 1.4 and still, are apparently very dangerous for Servlet Containers, Tomcat included. ThreadLocals are implemented as follows: The Thread object contains a hashmap whose Entries have weak key and hard value. The key is the ThreadLocal instance, and the value is the actual referent (the object returned by threadLocal.get()). The ThreadLocal instance confers this map for finding its value for the current Thread. This can be done unsyncronized, since it's obviously singlethreaded (its a map within the Thread!). Thus, when a ThreadLocal instance goes out of scope, its key will be GCable, which due to the coding of ThreadLocal at some point later, hopefully, will release the value (it "scavenges" dead keys on subsequent sets (of any other TL instance)). Now, ThreadLocals instances are often declarder as a "static (final)", right? In situations where the ClassLoader is ditched, while the Threads are recycled, this _instantly_ creates an enourmous leak: The ThreadLocalMap instance within the Thread contains an Entry with a weak reference to the ThreadLocal static instance, and a hard reference to the value. The value has a hard reference to its ClassLoader. The ClassLoader have a hard reference to the Class that contains the ThreadLocal static instance - hence the key is still hard reachable, and thus the key won't be nulled and GC'ed - and thus again the Thread.ThreadLocalMap.Entry instance won't ever be scavenged, and the Thread will forever after contain a hard reference to the now-not-used ClassLoader. (- which as explained above holds all the WebApp's classes, which have static fields, which point to structures within your web application, and you're screwed). Check out "Tackline"'s blog entry, which is the source of this information (I've hopefully understood his analysis!) http://www.jroller.com/page/tackline?entry=fixing_threadlocal This fine page also mentions ThreadLocals vs. Tomcat, but fails to point out that this should be viewed as a flaw with ThreadLocals: http://opensource2.atlassian.com/confluence/spring/pages/viewpage.action?pageId=2669 If you agree to this analysis, then vote for this bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6254531 ;-) Regards, Endre. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROT
Re: Memory leaking on [un|re]load: WebappClassLoader isn't GC'ed
On Jan 26, 2006, at 4:28 AM, Davide Romanini wrote: Il giorno mer, 25/01/2006 alle 11.12 -0800, Wade Chandler ha scritto: Don't use shared libraries and you shouldn't have this problem. If all of the statics are in your WEB-INF directory then you won't have an issue with unloading classes and class loaders. It would be great if could be so simple. But it is not, trust me, my shared folder is empty, and the common/lib folder contains only jdbc drivers and javamail implementation (needed by Tomcat if you want to use JNDI mail sessions). All the libraries of the webapp are in WEB-INF/ lib folder. As I said in previous mail, I tried to reload the distribution version of Axis (just as is, without other libs) and it causes the problem. It seems that something occuring during AxisServlet startup (it looks for his configuration using a complex discovery process) causes some class of a parent classloader to hold a reference to WebappClassloader so it cannot be GC'ed when it's reloaded. Davide, If you're using Axis, you might be interested in two Axis issues that I uncovered. They both caused ClassLoader memory leaks within Geronimo. The problems are described here: http://issues.apache.org/jira/browse/AXIS-2232 http://issues.apache.org/jira/browse/AXIS-2278 The changes are committed in SVN. However, there hasn't been an official Axis release, since then. The following binary contains the updates and is the version delivered by Geronimo: http://cvs.apache.org/repository/axis/jars/axis-1.4-356167.jar --kevan Surely it's not a Tomcat problem anyway... Bye, Davide Romanini - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory leaking on [un|re]load: WebappClassLoader isn't GC'ed
Il giorno gio, 26/01/2006 alle 06.23 -0500, Kevan Miller ha scritto: > Davide, > If you're using Axis, you might be interested in two Axis issues that > I uncovered. They both caused ClassLoader memory leaks within > Geronimo. The problems are described here: > http://issues.apache.org/jira/browse/AXIS-2232 > http://issues.apache.org/jira/browse/AXIS-2278 > > The changes are committed in SVN. However, there hasn't been an > official Axis release, since then. The following binary contains the > updates and is the version delivered by Geronimo: > > http://cvs.apache.org/repository/axis/jars/axis-1.4-356167.jar Thanks, I'll take a try. I hope to solve the problem. Bye, Davide Romanini - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
