James,
On 7/11/23 10:21, James Boggs wrote:
We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows
Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to
ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL
csr file, and imported a new SSL certificate from the CA into the new
keystore.
The website works but after logging in there are memory leak warnings
and the Tomcat service crashes within just a couple of minutes.
We even upgraded to 9.0.76 and the issue persists.
Below is an excerpt from the stderr log file.
I have been unable to find any recent threads on this, any help is
appreciated. Is any other information needed?
I think you have included all necessary information. I'm chopping-out
the irrrelevant bits:
2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to
unregister it when the web application was stopped. To prevent a memory
leak, the JDBC Driver has been forcibly unregistered.
This is actually "okay" in that Tomcat has detected a global JDBC driver
registration performed by the application, and is fixing the problem for
you. The application, however, is making a mistake by not de-registering
that JDBC driver. Your options are to move the driver library from your
application into Tomcat's lib/ directory, fix the application to
de-register the driver when it shuts down, or just ignore the warning.
But you should fix the application.
2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
appears to have started a thread named
[oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
There are multiple instances of this same message and THIS is your
problem. The problem is what the error message says: your application
has started a Thread and never stopped it. The "memory leak" comes from
the fact that the Thread has inherited the web application's ClassLoader
and the web application is being re-loaded. When that happens, Tomcat
discards the ClassLoader which usually means the GC will clean up after
it at some point in the future. But that Thread is still running and
will keep the ClassLoader in memory, likely forever.
You have a few options:
1. Fix the application. The application needs to shut-down any threads
is starts during its operation, preferrably in a
ServletContextListener's destroy method or similar.
2. Don't hot-reload your application. Instead, shut-down the JVM and
re-start it. Ovbviously, this may have availability implications, but
then again so does running out of memory and having to bounce the JVM,
anyway.
2023-07-10T21:35:40.944Z SEVERE The web application [rplans-vpd]
created a ThreadLocal with key of type
[java.lang.ThreadLocal.SuppliedThreadLocal] (value
[java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of
type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to
remove it when the web application was stopped. Threads are going to be
renewed over time to try and avoid a probable memory leak.
This is actually "okay" in that Tomcat has detected your application's
ThreadLocal variables (objects bound to one or more Threads which are
owned by the application) which are not being cleaned-up by the
application, and it's fixing the problem for you by, over time, killing
each of those Threads and replacing them in the Thread pool for you.
Your options are to fix the application or to ignore the warning. But
you should fix the application.
It appears that your upgrade of ORDS has introduced a lot of stuff that
doesn't play well with hot-reloading of the application. I'm assuming
that you aren't responsible for maintaining ORDS... Oracle is.
You should report all of the previous issues to Oracle against their
ORDS version 22.1 and ask them to fix them. It's why you write those
big, fat checks in the first place ;)
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
