Re: [OT] Tomcat 7 parallel deployment and PermGen Heap Space
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mark, On 7/20/2011 4:59 PM, Mark Thomas wrote: On 20/07/2011 21:46, Christopher Schultz wrote: Mark, On 7/20/2011 4:25 AM, Mark Thomas wrote: You need to de-register the driver you registered. However, that won't be causing the leak since Tomcat already did the clean up for you. Time to get the profiler out. Quick question: I get these JDBC Driver messages from Tomcat when I undeploy, but my code does not register the driver itself... The code might not, but registration is triggered by the application including the JAR. Just putting a JAR with a JDBC driver and the appropriate content in META-INF/services is enough to trigger registration with the JVM. Tomcat has nothing to do with this. I was about to boldly state that I didn't have a JDBC driver in my webapp's lib directory. Fortunately for me, I triple-checked first to make sure that wasn't the case and, for some reason, the driver had snuck into that directory. In my webapp's source, the JAR file was sitting there participating in the build process, yet not under source control. I must have done something stupid at some point. Fortunately, this is dev and not prod :) So, I have saved myself the embarrassment of stating that my webapp /absolutely does not/ have a JDBC driver in it, opting for the equally embarrassing I should have checked my setup before publicly complaining about Tomcat's behavior :) Tomcat does via a Resource defined in context.xml. Is that driver registered with the WebappClassLoader and therefore considered registered by the webapp? No. It is registered with the DriverManager. It is loaded by the WebappClassLoader which triggers the potential memory leak. Er, that's what I meant. I've been irritated by this behavior for a while because I feel like the container is performing the registration of the driver, but expecting the webapp to perform the de-registration. The META-INF/services mechanism does not provide automatic de-registration so the app has to take care of it. I don't use META-INF/services but it's obvious what the problem is, here. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4toxYACgkQ9CaO5/Lv0PDMpwCgpHgHx1c9APsMLNirnetR7/p+ KEYAnilC8kwpxrJ74ffWtJvlysP+J3C/ =5pO+ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Tomcat 7 parallel deployment and PermGen Heap Space
My personal advice with the Oracle driver is to put it at the server level, not at the webapp level. For instance if you use the query timeouts, oracle JDBC spawns a Thread to handle the timeouts, and there is no way to properly stop this thread so tomcat will continue complaining about leaking a classloader... With the driver at the server level, the warnings will disappear and your webapp won't leak (or at least not because of the JDBC driver). Sylvain On 20 juil. 2011, at 22:59, Mark Thomas wrote: On 20/07/2011 21:46, Christopher Schultz wrote: Mark, On 7/20/2011 4:25 AM, Mark Thomas wrote: You need to de-register the driver you registered. However, that won't be causing the leak since Tomcat already did the clean up for you. Time to get the profiler out. Quick question: I get these JDBC Driver messages from Tomcat when I undeploy, but my code does not register the driver itself... The code might not, but registration is triggered by the application including the JAR. Just putting a JAR with a JDBC driver and the appropriate content in META-INF/services is enough to trigger registration with the JVM. Tomcat has nothing to do with this. Tomcat does via a Resource defined in context.xml. Is that driver registered with the WebappClassLoader and therefore considered registered by the webapp? No. It is registered with the DriverManager. It is loaded by the WebappClassLoader which triggers the potential memory leak. I've been irritated by this behavior for a while because I feel like the container is performing the registration of the driver, but expecting the webapp to perform the de-registration. The META-INF/services mechanism does not provide automatic de-registration so the app has to take care of it. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 7 parallel deployment and PermGen Heap Space
On 20/07/2011 05:26, Monsieur fsfu wrote: @Chuck -- The moment I remove context xml (xyz##001.xml) file, tomcat automagically removes the corresponding dir from webapps. So that's not an issue. In my log file i see the following message INFO | jvm 1| 2011/07/19 16:11:07 | Jul 19, 2011 4:11:07 PM org.apache.catalina.startup.HostConfig checkResources INFO | jvm 1| 2011/07/19 16:11:07 | INFO: Undeploying context [##001] INFO | jvm 1| 2011/07/19 16:11:07 | Jul 19, 2011 4:11:07 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc INFO | jvm 1| 2011/07/19 16:11:07 | SEVERE: The web application [##001] registered the JDBC driver [oracle.jdbc.driver.OracleDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. Currently i am using commons-dbcp connection pooling. I will switch to Tomcat-JDBC connection pool and see if memory leak message goes away or not. I will update this thread with my findings. It won't. You need to de-register the driver you registered. However, that won't be causing the leak since Tomcat already did the clean up for you. Time to get the profiler out. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Tomcat 7 parallel deployment and PermGen Heap Space
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mark, On 7/20/2011 4:25 AM, Mark Thomas wrote: You need to de-register the driver you registered. However, that won't be causing the leak since Tomcat already did the clean up for you. Time to get the profiler out. Quick question: I get these JDBC Driver messages from Tomcat when I undeploy, but my code does not register the driver itself... Tomcat does via a Resource defined in context.xml. Is that driver registered with the WebappClassLoader and therefore considered registered by the webapp? I've been irritated by this behavior for a while because I feel like the container is performing the registration of the driver, but expecting the webapp to perform the de-registration. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4nPr8ACgkQ9CaO5/Lv0PCtawCglncNAXh36PhsbX7RumBEY7UK 8OgAni40dlStbgDSEtskmROivDqPvO0b =9j/G -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Tomcat 7 parallel deployment and PermGen Heap Space
On 20/07/2011 21:46, Christopher Schultz wrote: Mark, On 7/20/2011 4:25 AM, Mark Thomas wrote: You need to de-register the driver you registered. However, that won't be causing the leak since Tomcat already did the clean up for you. Time to get the profiler out. Quick question: I get these JDBC Driver messages from Tomcat when I undeploy, but my code does not register the driver itself... The code might not, but registration is triggered by the application including the JAR. Just putting a JAR with a JDBC driver and the appropriate content in META-INF/services is enough to trigger registration with the JVM. Tomcat has nothing to do with this. Tomcat does via a Resource defined in context.xml. Is that driver registered with the WebappClassLoader and therefore considered registered by the webapp? No. It is registered with the DriverManager. It is loaded by the WebappClassLoader which triggers the potential memory leak. I've been irritated by this behavior for a while because I feel like the container is performing the registration of the driver, but expecting the webapp to perform the de-registration. The META-INF/services mechanism does not provide automatic de-registration so the app has to take care of it. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat 7 parallel deployment and PermGen Heap Space
Hi,
I was checking out the parallel deployment feature of tomcat 7 and
encountered an issue with PermGen space of JVM. After the initial
deployment (xyz##001.war), PermGen space reaches ~120mb of space and
after parallel deployment (xyz##002.war) it almost doubles (~205mb)
which make sense. Now if i remove the old war file (xyz##001.war) by
removing xyz##001.xml
($catalina_home/conf/Catalina/locahost/xyz##001.xml), PermGen space
doesn't come down. I have waited upto 4 hrs but no luck. Once i
restart the JVM then only it comes down to ~120 range. Is anybody else
faced this issue? I would hate to restart the JVM after deployment
which totally defeats the purpose of parallel deployment.
Setup details:
OS -- RHEL - 5.5 (64 Bit)
Tomcat -- 7.0.14
Java -- jre1.6.0_25 (64 bit)
Tomcat APR -- apr-1.4.5 (64 bit)
java service wrapper -- 3.5.9 (64 bit)
Attached are server.xml and wrapper.conf files.
Thanks in advance
Kapil
?xml version='1.0' encoding='utf-8'?
Server port=8005 shutdown=tomcat7shutdown
Listener className=org.apache.catalina.security.SecurityListener /
Listener className=org.apache.catalina.core.AprLifecycleListener
SSLEngine=on /
Listener className=org.apache.catalina.core.JasperListener /
Listener
className=org.apache.catalina.core.JreMemoryLeakPreventionListener /
Listener
className=org.apache.catalina.mbeans.GlobalResourcesLifecycleListener /
Listener
className=org.apache.catalina.core.ThreadLocalLeakPreventionListener /
!-- Global JNDI resources --
GlobalNamingResources
Environment name=config/envType override=false type=java.lang.String
value=DEV2/
Environment name=config/envLocation override=false
type=java.lang.String value=SD/
Resource name=UserDatabase auth=Container
type=org.apache.catalina.UserDatabase
description=User database that can be updated and saved
factory=org.apache.catalina.users.MemoryUserDatabaseFactory
pathname=conf/tomcat-users.xml /
Resource name=UserDatabase auth=Container
type=org.apache.catalina.UserDatabase
description=User database that can be updated and saved
factory=org.apache.catalina.users.MemoryUserDatabaseFactory
pathname=conf/tomcat-users.xml /
JNDI Detail
/GlobalNamingResources
Service name=Catalina
Connector protocol=org.apache.coyote.http11.Http11AprProtocol
port=8080 maxHttpHeaderSize=8192
maxThreads=150 minSpareThreads=15 maxSpareThreads=50
enableLookups=false disableUploadTimeout=true
maxPostSize=8388608
acceptCount=100 connectionTimeout=2 /
Connector protocol=org.apache.coyote.http11.Http11AprProtocol
port=8443 maxHttpHeaderSize=8192
maxThreads=150 minSpareThreads=15 maxSpareThreads=50
enableLookups=false disableUploadTimeout=true
maxPostSize=8388608
acceptCount=100 connectionTimeout=2
scheme=https secure=true
clientAuth=false sslProtocol=TLS
SSLEnabled=true
SSLCertificateFile=${catalina.base}/conf/default.cert
SSLCertificateKeyFile=${catalina.base}/conf/default.key /
Engine name=Catalina defaultHost=localhost jvmRoute=jvm1
Realm className=org.apache.catalina.realm.LockOutRealm
Realm className=org.apache.catalina.realm.UserDatabaseRealm
resourceName=UserDatabase/
/Realm
Host name=localhost appBase=webapps
unpackWARs=true autoDeploy=true
xmlValidation=false xmlNamespaceAware=false
Valve className=org.apache.catalina.valves.AccessLogValve
directory=logs prefix=access suffix=.log
rotatable=false
pattern=%h %l %u %t %r %s %b %m %T %I resolveHosts=false/
/Host
/Engine
/Service
/Server
#encoding=UTF-8
#
# Wrapper Java Properties
#
# Java Application
# Locate the java binary on the system PATH:
wrapper.java.command=/local/mnt/java/jre1.6.0_25/bin/java
# Tell the Wrapper to log the full generated Java command line.
#wrapper.java.command.loglevel=INFO
# Java Main class. This class must implement the WrapperListener interface
# or guarantee that the WrapperManager class is initialized. Helper
# classes are provided to do this for you. See the Integration section
# of the documentation for details.
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperStartStopApp
#wrapper.java.mainclass=org.tanukisoftware.wrapper.test.Main
# Java Classpath (include wrapper.jar) Add class path elements as
# needed starting from 1
wrapper.java.classpath.1=../lib/wrappertest.jar
wrapper.java.classpath.2=../lib/wrapper.jar
wrapper.java.classpath.3=../bin/tomcat-juli.jar
Re: Tomcat 7 parallel deployment and PermGen Heap Space
On 20.07.2011 01:32, Monsieur fsfu wrote: Hi, I was checking out the parallel deployment feature of tomcat 7 and encountered an issue with PermGen space of JVM. After the initial deployment (xyz##001.war), PermGen space reaches ~120mb of space and after parallel deployment (xyz##002.war) it almost doubles (~205mb) which make sense. Now if i remove the old war file (xyz##001.war) by removing xyz##001.xml ($catalina_home/conf/Catalina/locahost/xyz##001.xml), PermGen space doesn't come down. I have waited upto 4 hrs but no luck. Once i restart the JVM then only it comes down to ~120 range. Is anybody else faced this issue? I would hate to restart the JVM after deployment which totally defeats the purpose of parallel deployment. Setup details: OS -- RHEL - 5.5 (64 Bit) Tomcat -- 7.0.14 Java -- jre1.6.0_25 (64 bit) Tomcat APR -- apr-1.4.5 (64 bit) java service wrapper -- 3.5.9 (64 bit) Attached are server.xml and wrapper.conf files. Look for the word leak in the Tomcat log files after undeploying one version. Those log lines give you hints, why Tomcat could not unload the classes. Unfortunately the solutions can be tricky, so for complex web applications it is not uncommon, that you migt have to live with those leaks, in other words provide more perm gen space and still restart every now and then. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 7 parallel deployment and PermGen Heap Space
2011/7/20 Monsieur fsfu monsieurf...@gmail.com: I was checking out the parallel deployment feature of tomcat 7 and encountered an issue with PermGen space of JVM. After the initial deployment (xyz##001.war), PermGen space reaches ~120mb of space and after parallel deployment (xyz##002.war) it almost doubles (~205mb) which make sense. Now if i remove the old war file (xyz##001.war) by removing xyz##001.xml ($catalina_home/conf/Catalina/locahost/xyz##001.xml), PermGen space doesn't come down. I have waited upto 4 hrs but no luck. PermGen usage is caused by classloader of the old webapp that is still present in memory. There can be many causes of that. You would have to take a heap dump and analyze it using a tool like MAT from Eclipse IDE. You may read http://wiki.apache.org/tomcat/MemoryLeakProtection Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 7 parallel deployment and PermGen Heap Space
From: Monsieur fsfu [mailto:monsieurf...@gmail.com] Subject: Tomcat 7 parallel deployment and PermGen Heap Space Now if i remove the old war file (xyz##001.war) by removing xyz##001.xml ($catalina_home/conf/Catalina/locahost/xyz##001.xml), PermGen space doesn't come down. First, verify that the first instance really has been undeployed. Second, unless you insure that at least two full GCs happen after undeploying the first .war file, class unloading will not occur. Even then, it's a bit of guess whether or not the JVM will choose to unload classes in the absence of pressure on PermGen. Regardless, as others suggested, running some heap analysis tool will find out if you've got a leak built into your webapp, or if the JVM has simply decided not to bother with class unloading yet. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 7 parallel deployment and PermGen Heap Space
@Chuck -- The moment I remove context xml (xyz##001.xml) file, tomcat automagically removes the corresponding dir from webapps. So that's not an issue. In my log file i see the following message INFO | jvm 1| 2011/07/19 16:11:07 | Jul 19, 2011 4:11:07 PM org.apache.catalina.startup.HostConfig checkResources INFO | jvm 1| 2011/07/19 16:11:07 | INFO: Undeploying context [##001] INFO | jvm 1| 2011/07/19 16:11:07 | Jul 19, 2011 4:11:07 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc INFO | jvm 1| 2011/07/19 16:11:07 | SEVERE: The web application [##001] registered the JDBC driver [oracle.jdbc.driver.OracleDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. Currently i am using commons-dbcp connection pooling. I will switch to Tomcat-JDBC connection pool and see if memory leak message goes away or not. I will update this thread with my findings. Thanks all you guys with your suggestion... On Tue, Jul 19, 2011 at 5:33 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Monsieur fsfu [mailto:monsieurf...@gmail.com] Subject: Tomcat 7 parallel deployment and PermGen Heap Space Now if i remove the old war file (xyz##001.war) by removing xyz##001.xml ($catalina_home/conf/Catalina/locahost/xyz##001.xml), PermGen space doesn't come down. First, verify that the first instance really has been undeployed. Second, unless you insure that at least two full GCs happen after undeploying the first .war file, class unloading will not occur. Even then, it's a bit of guess whether or not the JVM will choose to unload classes in the absence of pressure on PermGen. Regardless, as others suggested, running some heap analysis tool will find out if you've got a leak built into your webapp, or if the JVM has simply decided not to bother with class unloading yet. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
