RE: Tomcat 7 Web Service Shutdown Can't Stop Threads
Here is the code to shut down the pools. When I was running a single pool the
issue did not occur. It's when I added the second pool that the issue started.
Arrays.asList(universalConnectionPoolManager.getConnectionPoolNames())
.forEach(poolName -> {
try {
universalConnectionPoolManager
.destroyConnectionPool(poolName);
} catch (Exception e) {
exceptionHandler.handleError(
e,
String.join(
":",
"Destroy Connection Pool Error",
poolName));
}
});
The jdbc driver and connection pool .jar files are in the tomcat lib folder and
not in the applications lib folder.
I never tried Tomcat's version of connection pooling.
Thank you,
Chris Brown
Systems Consultant
National Council on
Compensation Insurance
901 Peninsula Corporate Circle
Boca Raton, FL 33487-1362
(P) 561-893-2324
chris_br...@ncci.com
-Original Message-
From: Christopher Schultz [mailto:ch...@christopherschultz.net]
Sent: Monday, March 07, 2016 10:33 AM
To: Tomcat Users List
Subject: Re: Tomcat 7 Web Service Shutdown Can't Stop Threads
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Chris,
On 3/7/16 9:51 AM, Chris Brown wrote:
> Tomcat is shutting down the web service when I deploy a new .war file.
Okay. You said you were closing the connection pool, though:
On 3/4/16 2:13 PM, Chris Brown wrote:
> I added a ServletContextListener to our web service. On the
> contextDestroyed method we destroy the connection pools. Our web
> service uses 2 connection pools.
So, how, precisely, are you shutting those pools down?
> There is no documentation on the threads and what they are for in
> oracle or tomcats documentation.
Tomcat would not contain information on a Vendor's product. A quick Google
search for "oracle ojdbc7 ucp-worker-thread" yields the following very relevant
result:
http://stackoverflow.com/questions/9671270/oracle-ucp-driver-and-tomcat-
threads-failing-to-stop
Also, searching for the OJDBC manual has a FAQ which mentions threads and their
lifetimes:
http://www.oracleimg.com/technetwork/articles/sql/jdbc-faq-090281.html
> The threads are not shutting down and they stay alive when the web
> service is shut down. I know this because I count the number of
> threads each time the web service context is destroyed and number
> keeps growing for each deployment of the .war file.
Oh, I believe you that they are piling up.
Is there a particular reason you are using Oracle's connection pool rather than
the one provided by the container (Tomcat)? Tomcat's pool won't leak like this
one does. Then again, the Oracle driver itself might still leak, but it will be
less leaky.
It's possible that the driver needs to be loaded before the application in
order to avoid pinning the ClassLoader into memory.
That can be done with Tomcat's configuration, but I'd only recommend that you
do this if nothing else is working.
I'd still like to know that those other threads (e.g. Thread-0) are doin g.
- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlbdnygACgkQ9CaO5/Lv0PCClgCfUmDNGz68sT6beiMOwx2Pkewb
UGcAoJNpSVJ8wagZRiGZMS9LxO3g0KY9
=qWGg
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
The information contained in this e-mail message is intended only for the
personal and confidential use of the recipient(s) named above. This message may
be an attorney-client communication and/or work product and as such is
privileged and confidential. If the reader of this message is not the intended
recipient or an agent responsible for delivering it to the intended recipient,
you are hereby notified that you have received this document in error and that
any review, dissemination, distribution, or copying of this message is strictly
prohibited. If you have received this communication in error, please notify us
immediately by e-mail, and delete the original message.
RE: Tomcat 7 Web Service Shutdown Can't Stop Threads
Tomcat is shutting down the web service when I deploy a new .war file. There is no documentation on the threads and what they are for in oracle or tomcats documentation. The threads are not shutting down and they stay alive when the web service is shut down. I know this because I count the number of threads each time the web service context is destroyed and number keeps growing for each deployment of the .war file. Thank you, Chris Brown Systems Consultant National Council on Compensation Insurance 901 Peninsula Corporate Circle Boca Raton, FL 33487-1362 (P) 561-893-2324 chris_br...@ncci.com -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Monday, March 07, 2016 9:47 AM To: Tomcat Users List Subject: Re: Tomcat 7 Web Service Shutdown Can't Stop Threads -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chris, On 3/7/16 8:26 AM, Chris Brown wrote: > The .jar comes with the Oracle 12c client. Oracle JDBC Drivers > release 12.1.0.2.0 Thanks. > I upgraded to tomcat 8. The latest stable version. The issue still > occurred, but the messages in the logs were "Warnings" > instead of being "SEVERE" errors. You said that. How about taking a thread dump after your web application stops, to see a few things: 1. Whether this is a spurious notification, since Tomcat was just going down faster than the threads were able to (gracefully) shut down 2. What those threads are doing, especially the ones with the somewhat anonymous names You haven't said how you were shutting-down the thread pool. You also haven't said whether you read any or Oracle's documentation about what those threads are doing or how they should be terminated. - -chris > -Original Message- From: Christopher Schultz > [mailto:ch...@christopherschultz.net] Sent: Friday, March 04, 2016 > 4:33 PM To: Tomcat Users List Subject: Re: Tomcat 7 Web Service > Shutdown Can't Stop Threads > > Chris, > > On 3/4/16 2:13 PM, Chris Brown wrote: >> Tomcat version: 7.0.42 Server OS: Sun Solaris UCP Version: Oracle 12c >> Oracle Driver: ojdbc7 > > That doesn't look like an Oracle JDBC driver version I've seen before. > Can you clarify? > >> JDK version: 1.8.0_31 > > You might want to upgrade Tomcat and Java. Those are somewhat old. > >> Issue Description: When tomcat shuts down our web service during a >> hot deployment threads are left running that tomcat cannot shutdown. >> I believe the threads are from the universal connection pool. We use >> oracle's universal connection pool (ucp.jar). After multiple >> deployments the threads build up and cause high CPU usage from the 2 >> tomcat threads. > >> Log Error Examples: Feb 08, 2016 8:00:25 PM >> org.apache.catalina.loader.WebappClassLoader >> clearReferencesThreads SEVERE: The web application [/RatingService] >> appears to have started a thread named [UCP-worker-thread-3] but has >> failed to stop it. This is very likely to create a memory leak. Feb >> 08, 2016 8:00:25 PM org.apache.catalina.loader.WebappClassLoader >> clearReferencesThreads SEVERE: The web application [/RatingService] >> appears to have started a thread named [Thread-438] but has failed to >> stop it. This is very likely to create a memory leak. Feb 27, 2016 >> 6:15:14 PM org.apache.catalina.loader.WebappClassLoader >> clearReferencesThreads SEVERE: The web application [/RatingService] >> appears to have started a thread named >> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockRelea s > >> er] >> but has failed to stop it. This is very likely to create a >> memory leak. Feb 27, 2016 6:15:14 PM >> org.apache.catalina.loader.WebappClassLoader >> clearReferencesThreads SEVERE: The web application >> [/RatingService] appears to have started a thread named >> [ForkJoinPool.commonPool-worker-32] but has failed to stop it. >> This is very likely to create a memory leak. Feb 27, 2016 6:15:14 >> PM org.apache.catalina.loader.WebappClassLoader >> clearReferencesThreads SEVERE: The web application >> [/RatingService] appears to have started a thread named >> [Timer-121] but has failed to stop it. This is very likely to >> create a memory leak. > >> Correction made to code to try and fix issue: I added a >> ServletContextListener to our web service. On the >> contextDestroyed method we destroy the connection pools. Our web >> service uses 2 connection pools. > > How exactly are you destroying the connection pools? > >> I also upgraded to Tomcat 8 and the issue still occurred, but >> the messages are now warnings instead of SEVERE errors. > > That's just an effect of Tomcat downgrading the log level under > which it log
RE: Tomcat 7 Web Service Shutdown Can't Stop Threads
The .jar comes with the Oracle 12c client. Oracle JDBC Drivers release 12.1.0.2.0 I upgraded to tomcat 8. The latest stable version. The issue still occurred, but the messages in the logs were "Warnings" instead of being "SEVERE" errors. Thank you, Chris Brown Systems Consultant National Council on Compensation Insurance 901 Peninsula Corporate Circle Boca Raton, FL 33487-1362 (P) 561-893-2324 chris_br...@ncci.com -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Friday, March 04, 2016 4:33 PM To: Tomcat Users List Subject: Re: Tomcat 7 Web Service Shutdown Can't Stop Threads -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chris, On 3/4/16 2:13 PM, Chris Brown wrote: > Tomcat version: 7.0.42 Server OS: Sun Solaris UCP Version: Oracle 12c > Oracle Driver: ojdbc7 That doesn't look like an Oracle JDBC driver version I've seen before. Can you clarify? > JDK version: 1.8.0_31 You might want to upgrade Tomcat and Java. Those are somewhat old. > Issue Description: When tomcat shuts down our web service during a hot > deployment threads are left running that tomcat cannot shutdown. I > believe the threads are from the universal connection pool. We use > oracle's universal connection pool (ucp.jar). After multiple > deployments the threads build up and cause high CPU usage from the 2 > tomcat threads. > > Log Error Examples: Feb 08, 2016 8:00:25 PM > org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads SEVERE: The web application [/RatingService] > appears to have started a thread named [UCP-worker-thread-3] but has > failed to stop it. This is very likely to create a memory leak. Feb > 08, 2016 8:00:25 PM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads SEVERE: The web application [/RatingService] > appears to have started a thread named [Thread-438] but has failed to > stop it. This is very likely to create a memory leak. Feb 27, > 2016 6:15:14 PM org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads SEVERE: The web application [/RatingService] > appears to have started a thread named > [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleas er] > but has failed to stop it. This is very likely to create a memory > leak. Feb 27, 2016 6:15:14 PM > org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads SEVERE: The web application [/RatingService] > appears to have started a thread named > [ForkJoinPool.commonPool-worker-32] but has failed to stop it. This is > very likely to create a memory leak. Feb 27, 2016 6:15:14 PM > org.apache.catalina.loader.WebappClassLoader > clearReferencesThreads SEVERE: The web application [/RatingService] > appears to have started a thread named [Timer-121] but has failed to > stop it. This is very likely to create a memory leak. > > Correction made to code to try and fix issue: I added a > ServletContextListener to our web service. On the contextDestroyed > method we destroy the connection pools. Our web service uses 2 > connection pools. How exactly are you destroying the connection pools? > I also upgraded to Tomcat 8 and the issue still occurred, but the > messages are now warnings instead of SEVERE errors. That's just an effect of Tomcat downgrading the log level under which it logs those issues. > I've searched the web and have tried many suggestions that have not > helped except destroying the connection pools, but the issue still > occurs. Please help!!! After you get those messages in the log, if you do a thread dump on the JVM, do you still see those threads running? Generally speaking, Tomcat should be able to shut down even with the above errors. Hopefully, none of those threads are non-daemon threads, meaning that they shouldn't stop the JVM from stopping once the JVM's main thread terminates. If your web application leaves those threads running after deployment, then multiple re-deployments without a Tomcat/JVM restart will likely result in a large memory leak. (Those threads will pin the web application's ClassLoader in memory along with all the classes it ever loaded, and these will pile-up until you have exhausted your permgen or other heap spaces). > [UCP-worker-thread-3] This is almost certainly something form Oracle's UCP > [Thread-438] This could be anything. A thread dump could tell you what it might be. > [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser ] Obviously, this is a thread from Oracle's JDBC driver. > [ForkJoinPool.commonPool-worker-32] Maybe something from UCP? Not sure. > [Timer-121] This is usually a thread started by TimerTask, a stock Java class. But any component may have created this thread. It might be worth searching your own code to see if a TimerTask is being launched and never shut-d
Tomcat 7 Web Service Shutdown Can't Stop Threads
Tomcat version: 7.0.42 Server OS: Sun Solaris UCP Version: Oracle 12c Oracle Driver: ojdbc7 JDK version: 1.8.0_31 Issue Description: When tomcat shuts down our web service during a hot deployment threads are left running that tomcat cannot shutdown. I believe the threads are from the universal connection pool. We use oracle's universal connection pool (ucp.jar). After multiple deployments the threads build up and cause high CPU usage from the 2 tomcat threads. Log Error Examples: Feb 08, 2016 8:00:25 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/RatingService] appears to have started a thread named [UCP-worker-thread-3] but has failed to stop it. This is very likely to create a memory leak. Feb 08, 2016 8:00:25 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/RatingService] appears to have started a thread named [Thread-438] but has failed to stop it. This is very likely to create a memory leak. Feb 27, 2016 6:15:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/RatingService] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Feb 27, 2016 6:15:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/RatingService] appears to have started a thread named [ForkJoinPool.commonPool-worker-32] but has failed to stop it. This is very likely to create a memory leak. Feb 27, 2016 6:15:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/RatingService] appears to have started a thread named [Timer-121] but has failed to stop it. This is very likely to create a memory leak. Correction made to code to try and fix issue: I added a ServletContextListener to our web service. On the contextDestroyed method we destroy the connection pools. Our web service uses 2 connection pools. I also upgraded to Tomcat 8 and the issue still occurred, but the messages are now warnings instead of SEVERE errors. I've searched the web and have tried many suggestions that have not helped except destroying the connection pools, but the issue still occurs. Please help!!! Thank you, Chris Brown Systems Consultant National Council on Compensation Insurance 901 Peninsula Corporate Circle Boca Raton, FL 33487-1362 (P) 561-893-2324 chris_br...@ncci.com The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
