Bugs item #973162, was opened at 2004-06-15 14:02 Message generated for change (Comment added) made by sflexus You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=376685&aid=973162&group_id=22866
Category: JBossServer Group: None Status: Open Resolution: Accepted Priority: 5 Submitted By: Chris Effenberger (pinguine) Assigned to: Scott M Stark (starksm) Summary: OutOfMemoryError Deployment Initial Comment: Java version: 1.4.2_01,Sun Microsystems Inc. Java VM: Java HotSpot(TM) Client VM 1.4.2_01-b06,Sun Microsystems Inc. Windows NT 4.0,x86, SP6 If I hotdeploy beans (more than 20-30) the following ERROR appears: 2004-06-15 12:46:58,472 DEBUG [org.jboss.ejb.EJBDeployer](ScannerThread :) create, Company.jar 2004-06-15 12:47:01,166 ERROR [org.jboss.deployment.MainDeployer](ScannerThread :) could not create deployment: file:/C:/jboss/server/default/deploy/Company.jar java.lang.OutOfMemoryError 2004-06-15 12:47:01,176 DEBUG [org.jboss.util.NestedThrowable](ScannerThread :) org.jboss.util.NestedThrowable.parentTraceEnabled=true 2004-06-15 12:47:01,176 DEBUG [org.jboss.util.NestedThrowable](ScannerThread :) org.jboss.util.NestedThrowable.nestedTraceEnabled=false 2004-06-15 12:47:01,176 DEBUG [org.jboss.util.NestedThrowable](ScannerThread :) org.jboss.util.NestedThrowable.detectDuplicateNesting=tr ue 2004-06-15 12:47:01,176 DEBUG [org.jboss.deployment.scanner.URLDeploymentScanner] (ScannerThread :) Failed to deploy: org.jboss.deployment.scanner.URLDeploymentScanner$De [EMAIL PROTECTED] url=file:/C:/jboss/server/default/deploy/Company.jar, deployedLastModified=0 } org.jboss.deployment.DeploymentException: Could not create deployment: file:/C:/jboss/server/default/deploy/Company.jar; - nested throwable: (java.lang.OutOfMemoryError) at org.jboss.deployment.DeploymentException.rethrowAsDep loymentException(DeploymentException.java:39) at org.jboss.deployment.MainDeployer.create (MainDeployer.java:806) at org.jboss.deployment.MainDeployer.deploy (MainDeployer.java:644) at org.jboss.deployment.MainDeployer.deploy (MainDeployer.java:608) at sun.reflect.GeneratedMethodAccessor20.invoke (Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke (Method.java:324) at org.jboss.mx.server.ReflectedDispatcher.dispatch (ReflectedDispatcher.java:60) at org.jboss.mx.server.Invocation.dispatch (Invocation.java:61) at org.jboss.mx.server.Invocation.dispatch (Invocation.java:53) at org.jboss.mx.server.Invocation.invoke (Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke (AbstractMBeanInvoker.java:185) at org.jboss.mx.server.MBeanServerImpl.invoke (MBeanServerImpl.java:473) at org.jboss.mx.util.MBeanProxyExt.invoke (MBeanProxyExt.java:176) at $Proxy7.deploy(Unknown Source) at org.jboss.deployment.scanner.URLDeploymentScanner.de ploy(URLDeploymentScanner.java:304) at org.jboss.deployment.scanner.URLDeploymentScanner.sc an(URLDeploymentScanner.java:478) at org.jboss.deployment.scanner.AbstractDeploymentScann er$ScannerThread.doScan (AbstractDeploymentScanner.java:201) at org.jboss.deployment.scanner.AbstractDeploymentScann er$ScannerThread.loop (AbstractDeploymentScanner.java:212) at org.jboss.deployment.scanner.AbstractDeploymentScann er$ScannerThread.run (AbstractDeploymentScanner.java:191) Caused by: java.lang.OutOfMemoryError 2004-06-15 12:47:01,176 DEBUG [org.jboss.deployment.scanner.URLDeploymentScanner] (ScannerThread :) Watch URL for: file:/C:/jboss/server/default/deploy/Company.jar -> file:/C:/jboss/server/default/deploy/Company.jar ---------------------------------------------------------------------- Comment By: Alexei Yudichev (sflexus) Date: 2004-10-03 13:15 Message: Logged In: YES user_id=345880 I've spent some time unvestigating the problem and so far discovered that (1) If default jbossweb-tomcat50.sar is installed, looks like we have only commons-logging problem. At least, with simplest ear/war, will try a real big application tomorrow. (2) If Jetty 5 is used, additional JspWriterImpl->...- >UnifiedClassloader3 chain remains (http://kanika.yi.org/ ~alexei/jbossleaks/refs-from-jasper5-PageContext-pool-to- UnifiedClassLoader3.html), also see links below. (3) If Jetty 4 is installed, there are some more leaks in Jasper, which I have not checked. I have submitted some bug reports related to this case: http://nagoya.apache.org/bugzilla/show_bug.cgi?id=31510 https://sourceforge.net/tracker/? func=detail&aid=1038066&group_id=7322&atid=107322 ---------------------------------------------------------------------- Comment By: Scott M Stark (starksm) Date: 2004-09-23 18:58 Message: Logged In: YES user_id=175228 I'm back to looking at this and I'll see what can be done to fix the leaks in the apache code. ---------------------------------------------------------------------- Comment By: Alexei Yudichev (sflexus) Date: 2004-09-20 11:13 Message: Logged In: YES user_id=345880 commons-logging is unlikely to be fixed in the nearest future (http://issues.apache.org/bugzilla/show_bug.cgi?id=31286), however, I think, a patched commons-logging could be bundled in jboss, the one that is using WeakHashMap for example instead of Hashtable. Seems that LogFactory is not subclassed anywhere in JBoss so there will be no problems with that. ---------------------------------------------------------------------- Comment By: Alexei Yudichev (sflexus) Date: 2004-09-16 14:27 Message: Logged In: YES user_id=345880 Just to avoid duplicate work, is anybody (Scott?) working on this or not? ---------------------------------------------------------------------- Comment By: Alexei Yudichev (sflexus) Date: 2004-09-10 12:19 Message: Logged In: YES user_id=345880 Tried Jetty5, which is using Jasper from Tomcat 5, and seems that previous reference from Jasper logger now exists no more (because there is no more Jasper logger in new Jasper). However, a new one detected, from some pool of JspContexts in org.apache.jasper.runtime.JspFactoryImpl, see http:// kanika.yi.org/~alexei/jbossleaks/refs-from-jasper5- PageContext-pool-to-UnifiedClassLoader3.html. ---------------------------------------------------------------------- Comment By: Alexei Yudichev (sflexus) Date: 2004-09-10 10:39 Message: Logged In: YES user_id=345880 added another case, http://kanika.yi.org/~alexei/jbossleaks/ refs-from-jasper-constantsl-to-UnifiedClassLoader3.html, which is again the jasper's DefaultLogger holding reference to ServletContext. The key place is jasper code is in org.apache.jasper.servlet. JspServlet.java line 122 for tomcat 4.1.30, init() method: Constants.jasperLog = new DefaultLogger(this.context); Shouldnt this static field be set to null in destroy() once it is initialized in init() ?! Constants.jasperLog = null; However, as I see, in tomcat5 they moved completely to commons-logging in Jasper, so this particular problem is probably solved, but not for me :( ---------------------------------------------------------------------- Comment By: Alexei Yudichev (sflexus) Date: 2004-09-09 19:15 Message: Logged In: YES user_id=345880 Sorry, forgot to add, I have downloaded jetty-4.2.21-jboss-3. 2.5 bundle from sf.net/projects/jetty and replaced tomcat's sar with jetty. I have always used Jetty with JBoss. ---------------------------------------------------------------------- Comment By: Scott M Stark (starksm) Date: 2004-09-09 19:12 Message: Logged In: YES user_id=175228 How is this jboss-3.2.5 if there is a jetty web container? ---------------------------------------------------------------------- Comment By: Alexei Yudichev (sflexus) Date: 2004-09-09 19:08 Message: Logged In: YES user_id=345880 OK I have installer jprofiler and spent quite some time to find references to UnifiedClassLoader3 of the application after it is undeployed. First, there was a hard reference from static field org.apache.commons.logger.LogFactory.factories of type HashTable, which is a cache of LogFactories where a key is a ClassLoader. I patched commons-logging to disable LogFactory caching, but still there was no effect. Second hard reference, which I was unable to remove right away, is: static field org.apache.commons.logging.Logger.loggers of type Hashtable, which is a Hashtable<String,Logger> -> Hashtable value of type DefaultLogger -> field ServletContext servletContext -> ... -> finally this servlet context's ClassLoader. Third hard reference I found is from org.apache.tool.ant. taskdefs.Execute -> ProcessDestroyer (extends Thread) -> Thread's context loader. This is probably because one of proceses that ANT launched is not finished, most likely this was jikes compiler. Fourth reference was from org.jboss.util.threadpool. BasicThreadPool to the same ant's ProcessDestroyer and further. I have published all reference graphs (except the first one) here: http://kanika.yi.org/~alexei/jbossleaks/ This is all so far, will continue tomorrow. ---------------------------------------------------------------------- Comment By: Adrian Brock (ejort) Date: 2004-09-09 12:01 Message: Logged In: YES user_id=9459 As Scott already said. The problem is due to (static) classes (that don't get redeployed) holding hard references to classloaders (either directly or indirectly through holding references to objects created from classes from that classloader) that have been undeployed. Some of the jakarta commons jars are notorious for this, but it is not exclusive to them. If the classloader doesn't get GCed, static classes loaded from that classloader won't either. Something not redeployed -> Object -> Class -> Undeployed ClassLoader -> Stuff Not Garbage Collected -> OutOfMemoryError You need to get hold of profiler that lets you see what is holding a hard reference to the classloader after it has been undeployed. If it is a JBoss class we will fix it. ---------------------------------------------------------------------- Comment By: Alexei Yudichev (sflexus) Date: 2004-09-09 11:07 Message: Logged In: YES user_id=345880 Cannot reopen it because it is not created by me. Below is a copy-paste of my comment to bug [ 576913 ] OutOfMemory after redeploys, showing steps to reproduce it by redeployment of EAR from JBoss testsuite. I have a question. If a redeploy an application, are its classes supposed to be reloaded and all static fields referencing other objects become subject to GC? I made a simple test by adding this code to one of my session bean classes: private static class GCChecker { public byte[] buf; public GCChecker() { buf = new byte[10 * 1024 * 1024]; System.out.println("[GCChecker] ALLOCATED 10Mb"); } protected void finalize() throws Throwable { System.out.println("[GCChecker] FINALIZED"); } } private final static GCChecker gcChecker = new GCChecker(); After 4 cycles of "bean usage attempt/app redeployment" I got OutOfMemoryException and in console there are only [GCChecker] ALLOCATED 10Mb messages. Is this a correct behaviour of class loaders and therefore I should set all static fields of all my classes to null at some application finalization stage? But then how about the class itself, how can I remove reference from classloader to loaded class? I'll copy the question to bug 576913 because I'm not sure if anybody is reading comments to closed bugs. This is how I reproduce it on my Windows system, step by step: 1) Download JBoss 3.2.5 sources, build a distribution (SRC_HOME/build/build.bat) and assume JBOSS_HOME=SRC_HOME/build/output/jboss-3.2.5 2) Execute build.bat build testsuite module (testsuite/build. bat), copy SRC_HOME/testsuite/output/lib/jbosstest-web.ear to JBOSS_HOME/server/all/deploy 3) change JBOSS_HOME/bin/run.bat to specify -Xmx40m in JVM parameters (just no to wait long). I also included -Xloggc: c:/temp/jboss-gc.log to monitor memory 4) start JBoss 5) execute the following ant script, changing jboss.home accordingly: <?xml version="1.0" encoding="ISO-8859-1"?> <project name="JBossOOMGenerator" default="redeploy- endlessly"> <property name="jboss.home" value="C:/temp/4/jboss-3.2. 5"/> <property name="deployment.url" value="${jboss.home}/ server/all/deploy/jbosstest-web.ear"/> <path id="jmx.task.classpath"> <pathelement location="${jboss.home}/client/jbossall- client.jar"/> <pathelement location="${jboss.home}/docs/examples/jmx/ jbossjmx-ant.jar"/> </path> <target name="redeploy-endlessly"> <taskdef name="jmx" classname="org.jboss.ant.JMX" classpathref="jmx.task.classpath"/> <antcall target="loop"/> </target> <target name="loop"> <jmx adapterName="jmx/rmi/RMIAdaptor"> <propertyeditor type="java.net.URL" editor="org.jboss.util. propertyeditor.URLEditor"/> <invoke target="jboss.system:service=MainDeployer" operation="redeploy"> <parameter type="java.net.URL" arg="${deployment.url} "/> </invoke> </jmx> <antcall target="loop2"/> </target> <target name="loop2"> <antcall target="loop"/> </target> </project> In approx. 30 minutes (on my P4 2.4GHz) I get OutOfMemoryError in JBoss JVM. I believe if I set the -Xmx to around 35m I would get it much sooner. ---------------------------------------------------------------------- Comment By: Scott M Stark (starksm) Date: 2004-09-07 21:34 Message: Logged In: YES user_id=175228 A trace level log is of no use here. You need to supply an example if you want use to look at it, or profile the redeployment yourself. This has been caused by static classes holding onto class loaders which in turn leak classes and data via caches in every case investigated so far. There are no known leaks in the jboss deployment layer at this time. Reopen with an example or steps to reproduce the issue. ---------------------------------------------------------------------- Comment By: Alexei Yudichev (sflexus) Date: 2004-09-07 17:59 Message: Logged In: YES user_id=345880 Welcome to the world of bugs #435958, #576913 and probably many others. ---------------------------------------------------------------------- Comment By: Chris Effenberger (pinguine) Date: 2004-06-15 14:35 Message: Logged In: YES user_id=733943 log with trace level is attached ---------------------------------------------------------------------- Comment By: Chris Effenberger (pinguine) Date: 2004-06-15 14:04 Message: Logged In: YES user_id=733943 jboss-3.2.4 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=376685&aid=973162&group_id=22866 ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ JBoss-Development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development