Suvendu,
On 7/12/23 07:11, Suvendu Sekhar Mondal wrote:
On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz
<ch...@christopherschultz.net> wrote:
James,
On 7/11/23 10:21, James Boggs wrote:
We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows
Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to
ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL
csr file, and imported a new SSL certificate from the CA into the new
keystore.
The website works but after logging in there are memory leak warnings
and the Tomcat service crashes within just a couple of minutes.
We even upgraded to 9.0.76 and the issue persists.
Below is an excerpt from the stderr log file.
I have been unable to find any recent threads on this, any help is
appreciated. Is any other information needed?
I think you have included all necessary information. I'm chopping-out
the irrrelevant bits:
2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to
unregister it when the web application was stopped. To prevent a memory
leak, the JDBC Driver has been forcibly unregistered.
This is actually "okay" in that Tomcat has detected a global JDBC driver
registration performed by the application, and is fixing the problem for
you. The application, however, is making a mistake by not de-registering
that JDBC driver. Your options are to move the driver library from your
application into Tomcat's lib/ directory, fix the application to
de-register the driver when it shuts down, or just ignore the warning.
But you should fix the application.
2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
appears to have started a thread named
[oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but
has failed to stop it. This is very likely to create a memory leak. Stack trace
of thread:
There are multiple instances of this same message and THIS is your
problem. The problem is what the error message says: your application
has started a Thread and never stopped it. The "memory leak" comes from
the fact that the Thread has inherited the web application's ClassLoader
and the web application is being re-loaded. When that happens, Tomcat
discards the ClassLoader which usually means the GC will clean up after
it at some point in the future. But that Thread is still running and
will keep the ClassLoader in memory, likely forever.
You have a few options:
1. Fix the application. The application needs to shut-down any threads
is starts during its operation, preferrably in a
ServletContextListener's destroy method or similar.
2. Don't hot-reload your application. Instead, shut-down the JVM and
re-start it. Ovbviously, this may have availability implications, but
then again so does running out of memory and having to bounce the JVM,
anyway.
I was checking the stacks and saw this: They might not be doing
"hot-deploy" of the app. AFAIK those messages come once someone stops
Tomcat.
2023-07-10T20:52:06.292Z INFO Starting ProtocolHandler
["https-openssl-nio-10.2.251.132-443"]
2023-07-10T20:52:06.346Z INFO Server startup in [28980] milliseconds
2023-07-10T21:35:40.487Z INFO Pausing ProtocolHandler
["https-openssl-nio-10.2.251.132-443"]
2023-07-10T21:35:40.495Z INFO Stopping service [Catalina]
2023-07-10T21:35:40.910Z INFO Destroyed pool :
|default|lo|-2023-07-10T20-51-53.099285500Z
Good point: Tomcat will complain about this stuff on shutdown even if
the warnings are irrelevant because the application-stop leak-checks are
run whether Tomcat is going to stop or not.
So operationally, these warnings may be irrelevant.
(IMHO you should still fix the application!)
Another thing I noticed was the difference in allocated Java heap. It
appears that first it sets min=max=1GB and after that it's setting max
heap to 256MB. That could be the problem. JVM might be crashing
because of heap shortage.
10-Jul-2023 16:51:35.481 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line argument:
-Dconfig.url=D:\ords222
10-Jul-2023 16:51:35.481 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line argument:
-Xms1024M
10-Jul-2023 16:51:35.481 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line argument:
-Xmx1024M
10-Jul-2023 16:51:35.486 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line argument:
exit
10-Jul-2023 16:51:35.486 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line argument:
abort
10-Jul-2023 16:51:35.486 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line argument:
-Xms128m
10-Jul-2023 16:51:35.486 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line argument:
-Xmx256m
Oh, I hadn't noticed that, either.
Dropping your heap to 25% of original values could certainly cause
problems :)
I don't see a "crash" in the log, so it's hard to tell what exactly is
being reported, here.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
