Bugs item #973162, was opened at 2004-06-15 14:02
Message generated for change (Comment added) made by sflexus
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=376685&aid=973162&group_id=22866
Category: None
Group: None
Status: Open
Resolution: Accepted
Priority: 5
Submitted By: Chris Effenberger (pinguine)
Assigned to: Scott M Stark (starksm)
Summary: OutOfMemoryError Deployment
Initial Comment:
Java version: 1.4.2_01,Sun Microsystems Inc.
Java VM: Java HotSpot(TM) Client VM 1.4.2_01-b06,Sun
Microsystems Inc.
Windows NT 4.0,x86, SP6
If I hotdeploy beans (more than 20-30) the following
ERROR appears:
2004-06-15 12:46:58,472 DEBUG
[org.jboss.ejb.EJBDeployer](ScannerThread :) create,
Company.jar
2004-06-15 12:47:01,166 ERROR
[org.jboss.deployment.MainDeployer](ScannerThread :)
could not create deployment:
file:/C:/jboss/server/default/deploy/Company.jar
java.lang.OutOfMemoryError
2004-06-15 12:47:01,176 DEBUG
[org.jboss.util.NestedThrowable](ScannerThread :)
org.jboss.util.NestedThrowable.parentTraceEnabled=true
2004-06-15 12:47:01,176 DEBUG
[org.jboss.util.NestedThrowable](ScannerThread :)
org.jboss.util.NestedThrowable.nestedTraceEnabled=false
2004-06-15 12:47:01,176 DEBUG
[org.jboss.util.NestedThrowable](ScannerThread :)
org.jboss.util.NestedThrowable.detectDuplicateNesting=tr
ue
2004-06-15 12:47:01,176 DEBUG
[org.jboss.deployment.scanner.URLDeploymentScanner]
(ScannerThread :) Failed to deploy:
org.jboss.deployment.scanner.URLDeploymentScanner$De
[EMAIL PROTECTED]
url=file:/C:/jboss/server/default/deploy/Company.jar,
deployedLastModified=0 }
org.jboss.deployment.DeploymentException: Could not
create deployment:
file:/C:/jboss/server/default/deploy/Company.jar; -
nested throwable: (java.lang.OutOfMemoryError)
at
org.jboss.deployment.DeploymentException.rethrowAsDep
loymentException(DeploymentException.java:39)
at org.jboss.deployment.MainDeployer.create
(MainDeployer.java:806)
at org.jboss.deployment.MainDeployer.deploy
(MainDeployer.java:644)
at org.jboss.deployment.MainDeployer.deploy
(MainDeployer.java:608)
at
sun.reflect.GeneratedMethodAccessor20.invoke
(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke
(Method.java:324)
at
org.jboss.mx.server.ReflectedDispatcher.dispatch
(ReflectedDispatcher.java:60)
at org.jboss.mx.server.Invocation.dispatch
(Invocation.java:61)
at org.jboss.mx.server.Invocation.dispatch
(Invocation.java:53)
at org.jboss.mx.server.Invocation.invoke
(Invocation.java:86)
at
org.jboss.mx.server.AbstractMBeanInvoker.invoke
(AbstractMBeanInvoker.java:185)
at org.jboss.mx.server.MBeanServerImpl.invoke
(MBeanServerImpl.java:473)
at org.jboss.mx.util.MBeanProxyExt.invoke
(MBeanProxyExt.java:176)
at $Proxy7.deploy(Unknown Source)
at
org.jboss.deployment.scanner.URLDeploymentScanner.de
ploy(URLDeploymentScanner.java:304)
at
org.jboss.deployment.scanner.URLDeploymentScanner.sc
an(URLDeploymentScanner.java:478)
at
org.jboss.deployment.scanner.AbstractDeploymentScann
er$ScannerThread.doScan
(AbstractDeploymentScanner.java:201)
at
org.jboss.deployment.scanner.AbstractDeploymentScann
er$ScannerThread.loop
(AbstractDeploymentScanner.java:212)
at
org.jboss.deployment.scanner.AbstractDeploymentScann
er$ScannerThread.run
(AbstractDeploymentScanner.java:191)
Caused by: java.lang.OutOfMemoryError
2004-06-15 12:47:01,176 DEBUG
[org.jboss.deployment.scanner.URLDeploymentScanner]
(ScannerThread :) Watch URL for:
file:/C:/jboss/server/default/deploy/Company.jar ->
file:/C:/jboss/server/default/deploy/Company.jar
----------------------------------------------------------------------
Comment By: Alexei Yudichev (sflexus)
Date: 2004-09-20 11:13
Message:
Logged In: YES
user_id=345880
commons-logging is unlikely to be fixed in the nearest future
(http://issues.apache.org/bugzilla/show_bug.cgi?id=31286),
however, I think, a patched commons-logging could be bundled
in jboss, the one that is using WeakHashMap for example
instead of Hashtable. Seems that LogFactory is not subclassed
anywhere in JBoss so there will be no problems with that.
----------------------------------------------------------------------
Comment By: Alexei Yudichev (sflexus)
Date: 2004-09-16 14:27
Message:
Logged In: YES
user_id=345880
Just to avoid duplicate work, is anybody (Scott?) working on
this or not?
----------------------------------------------------------------------
Comment By: Alexei Yudichev (sflexus)
Date: 2004-09-10 12:19
Message:
Logged In: YES
user_id=345880
Tried Jetty5, which is using Jasper from Tomcat 5, and seems
that previous reference from Jasper logger now exists no more
(because there is no more Jasper logger in new Jasper).
However, a new one detected, from some pool of JspContexts
in org.apache.jasper.runtime.JspFactoryImpl, see http://
kanika.yi.org/~alexei/jbossleaks/refs-from-jasper5-
PageContext-pool-to-UnifiedClassLoader3.html.
----------------------------------------------------------------------
Comment By: Alexei Yudichev (sflexus)
Date: 2004-09-10 10:39
Message:
Logged In: YES
user_id=345880
added another case, http://kanika.yi.org/~alexei/jbossleaks/
refs-from-jasper-constantsl-to-UnifiedClassLoader3.html,
which is again the jasper's DefaultLogger holding reference to
ServletContext.
The key place is jasper code is in org.apache.jasper.servlet.
JspServlet.java line 122 for tomcat 4.1.30, init() method:
Constants.jasperLog = new DefaultLogger(this.context);
Shouldnt this static field be set to null in destroy() once it is
initialized in init() ?!
Constants.jasperLog = null;
However, as I see, in tomcat5 they moved completely to
commons-logging in Jasper, so this particular problem is
probably solved, but not for me :(
----------------------------------------------------------------------
Comment By: Alexei Yudichev (sflexus)
Date: 2004-09-09 19:15
Message:
Logged In: YES
user_id=345880
Sorry, forgot to add, I have downloaded jetty-4.2.21-jboss-3.
2.5 bundle from sf.net/projects/jetty and replaced tomcat's
sar with jetty. I have always used Jetty with JBoss.
----------------------------------------------------------------------
Comment By: Scott M Stark (starksm)
Date: 2004-09-09 19:12
Message:
Logged In: YES
user_id=175228
How is this jboss-3.2.5 if there is a jetty web container?
----------------------------------------------------------------------
Comment By: Alexei Yudichev (sflexus)
Date: 2004-09-09 19:08
Message:
Logged In: YES
user_id=345880
OK I have installer jprofiler and spent quite some time to find
references to UnifiedClassLoader3 of the application after it is
undeployed. First, there was a hard reference from static field
org.apache.commons.logger.LogFactory.factories of type
HashTable, which is a cache of LogFactories where a key is a
ClassLoader. I patched commons-logging to disable LogFactory
caching, but still there was no effect.
Second hard reference, which I was unable to remove right
away, is:
static field org.apache.commons.logging.Logger.loggers of type
Hashtable, which is a Hashtable<String,Logger> ->
Hashtable value of type DefaultLogger -> field ServletContext
servletContext -> ... -> finally this servlet context's
ClassLoader.
Third hard reference I found is from org.apache.tool.ant.
taskdefs.Execute -> ProcessDestroyer (extends Thread) ->
Thread's context loader. This is probably because one of
proceses that ANT launched is not finished, most likely this
was jikes compiler.
Fourth reference was from org.jboss.util.threadpool.
BasicThreadPool to the same ant's ProcessDestroyer and
further.
I have published all reference graphs (except the first one)
here: http://kanika.yi.org/~alexei/jbossleaks/
This is all so far, will continue tomorrow.
----------------------------------------------------------------------
Comment By: Adrian Brock (ejort)
Date: 2004-09-09 12:01
Message:
Logged In: YES
user_id=9459
As Scott already said. The problem is due to (static) classes
(that don't get redeployed) holding hard references to
classloaders
(either directly or indirectly through holding references to
objects
created from classes from that classloader)
that have been undeployed.
Some of the jakarta commons jars are notorious for this,
but it is not exclusive to them.
If the classloader doesn't get GCed, static classes loaded
from that
classloader won't either.
Something not redeployed -> Object -> Class -> Undeployed
ClassLoader -> Stuff Not Garbage Collected -> OutOfMemoryError
You need to get hold of profiler that lets you see what is
holding a hard reference to the classloader after it has been
undeployed.
If it is a JBoss class we will fix it.
----------------------------------------------------------------------
Comment By: Alexei Yudichev (sflexus)
Date: 2004-09-09 11:07
Message:
Logged In: YES
user_id=345880
Cannot reopen it because it is not created by me. Below is a
copy-paste of my comment to bug [ 576913 ] OutOfMemory
after redeploys, showing steps to reproduce it by
redeployment of EAR from JBoss testsuite.
I have a question. If a redeploy an application, are its classes
supposed to be reloaded and all static fields referencing other
objects become subject to GC? I made a simple test by adding
this code to one of my session bean classes:
private static class GCChecker {
public byte[] buf;
public GCChecker() {
buf = new byte[10 * 1024 * 1024];
System.out.println("[GCChecker] ALLOCATED 10Mb");
}
protected void finalize() throws Throwable {
System.out.println("[GCChecker] FINALIZED");
}
}
private final static GCChecker gcChecker = new GCChecker();
After 4 cycles of "bean usage attempt/app redeployment" I
got OutOfMemoryException and in console there are only
[GCChecker] ALLOCATED 10Mb
messages.
Is this a correct behaviour of class loaders and therefore I
should set all static fields of all my classes to null at some
application finalization stage? But then how about the class
itself, how can I remove reference from classloader to loaded
class?
I'll copy the question to bug 576913 because I'm not sure if
anybody is reading comments to closed bugs.
This is how I reproduce it on my Windows system, step by
step:
1) Download JBoss 3.2.5 sources, build a distribution
(SRC_HOME/build/build.bat) and assume
JBOSS_HOME=SRC_HOME/build/output/jboss-3.2.5
2) Execute build.bat build testsuite module (testsuite/build.
bat), copy SRC_HOME/testsuite/output/lib/jbosstest-web.ear
to JBOSS_HOME/server/all/deploy
3) change JBOSS_HOME/bin/run.bat to specify -Xmx40m in
JVM parameters (just no to wait long). I also included -Xloggc:
c:/temp/jboss-gc.log to monitor memory
4) start JBoss
5) execute the following ant script, changing jboss.home
accordingly:
<?xml version="1.0" encoding="ISO-8859-1"?>
<project name="JBossOOMGenerator" default="redeploy-
endlessly">
<property name="jboss.home" value="C:/temp/4/jboss-3.2.
5"/>
<property name="deployment.url" value="${jboss.home}/
server/all/deploy/jbosstest-web.ear"/>
<path id="jmx.task.classpath">
<pathelement location="${jboss.home}/client/jbossall-
client.jar"/>
<pathelement location="${jboss.home}/docs/examples/jmx/
jbossjmx-ant.jar"/>
</path>
<target name="redeploy-endlessly">
<taskdef name="jmx" classname="org.jboss.ant.JMX"
classpathref="jmx.task.classpath"/>
<antcall target="loop"/>
</target>
<target name="loop">
<jmx adapterName="jmx/rmi/RMIAdaptor">
<propertyeditor type="java.net.URL"
editor="org.jboss.util.
propertyeditor.URLEditor"/>
<invoke target="jboss.system:service=MainDeployer"
operation="redeploy">
<parameter type="java.net.URL" arg="${deployment.url}
"/>
</invoke>
</jmx>
<antcall target="loop2"/>
</target>
<target name="loop2">
<antcall target="loop"/>
</target>
</project>
In approx. 30 minutes (on my P4 2.4GHz) I get
OutOfMemoryError in JBoss JVM. I believe if I set the -Xmx to
around 35m I would get it much sooner.
----------------------------------------------------------------------
Comment By: Scott M Stark (starksm)
Date: 2004-09-07 21:34
Message:
Logged In: YES
user_id=175228
A trace level log is of no use here. You need to supply an
example if you want use to look at it, or profile the
redeployment yourself. This has been caused by static
classes holding onto class loaders which in turn leak
classes and data via caches in every case investigated so
far. There are no known leaks in the jboss deployment layer
at this time. Reopen with an example or steps to reproduce
the issue.
----------------------------------------------------------------------
Comment By: Alexei Yudichev (sflexus)
Date: 2004-09-07 17:59
Message:
Logged In: YES
user_id=345880
Welcome to the world of bugs #435958, #576913 and probably
many others.
----------------------------------------------------------------------
Comment By: Chris Effenberger (pinguine)
Date: 2004-06-15 14:35
Message:
Logged In: YES
user_id=733943
log with trace level is attached
----------------------------------------------------------------------
Comment By: Chris Effenberger (pinguine)
Date: 2004-06-15 14:04
Message:
Logged In: YES
user_id=733943
jboss-3.2.4
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=376685&aid=973162&group_id=22866
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
JBoss-Development mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-development
