James,
On 7/12/23 15:41, James Boggs wrote:
Thanks for the input. I will forward the email to our developers to
look at the heap size settings being different.
We have a Windows service that is used to start/stop Tomcat. When
this happens we find that the Windows service is no longer running.
Depending upon your settings, you may have more than one log file to
look at. Look at everything in CATALINA_BASE/logs the next time the
service goes down to see if there is a clue in any of them.
You could also look for hs_pid[process-id] files (or the Windows
equivalent, if it's different) that are usually dumped when a JVM crash
occurs.
Each of your logs seems to show that Tomcat is being shut-down in an
orderly way, though. Are you sure it's crashing and not being
intentionally stopped?
-chris
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Wednesday, July 12, 2023 3:32 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat 9.0.76 Memory leak with Java 17
Suvendu,
On 7/12/23 07:11, Suvendu Sekhar Mondal wrote:
On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz
<ch...@christopherschultz.net> wrote:
James,
On 7/11/23 10:21, James Boggs wrote:
We had a stable SSL enabled website with Apache Tomcat 9.0.73 on
Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and
to ORDS 22.1, then used Java 17 to create a new Java Keystore and a
new SSL csr file, and imported a new SSL certificate from the CA
into the new keystore.
The website works but after logging in there are memory leak
warnings and the Tomcat service crashes within just a couple of minutes.
We even upgraded to 9.0.76 and the issue persists.
Below is an excerpt from the stderr log file.
I have been unable to find any recent threads on this, any help is
appreciated. Is any other information needed?
I think you have included all necessary information. I'm chopping-out
the irrrelevant bits:
2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to
unregister it when the web application was stopped. To prevent a
memory leak, the JDBC Driver has been forcibly unregistered.
This is actually "okay" in that Tomcat has detected a global JDBC
driver registration performed by the application, and is fixing the
problem for you. The application, however, is making a mistake by not
de-registering that JDBC driver. Your options are to move the driver
library from your application into Tomcat's lib/ directory, fix the
application to de-register the driver when it shuts down, or just ignore the
warning.
But you should fix the application.
2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
appears to have started a thread named
[oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but
has failed to stop it. This is very likely to create a memory leak. Stack trace
of thread:
There are multiple instances of this same message and THIS is your
problem. The problem is what the error message says: your application
has started a Thread and never stopped it. The "memory leak" comes
from the fact that the Thread has inherited the web application's
ClassLoader and the web application is being re-loaded. When that
happens, Tomcat discards the ClassLoader which usually means the GC
will clean up after it at some point in the future. But that Thread
is still running and will keep the ClassLoader in memory, likely forever.
You have a few options:
1. Fix the application. The application needs to shut-down any
threads is starts during its operation, preferrably in a
ServletContextListener's destroy method or similar.
2. Don't hot-reload your application. Instead, shut-down the JVM and
re-start it. Ovbviously, this may have availability implications, but
then again so does running out of memory and having to bounce the
JVM, anyway.
I was checking the stacks and saw this: They might not be doing
"hot-deploy" of the app. AFAIK those messages come once someone stops
Tomcat.
2023-07-10T20:52:06.292Z INFO Starting ProtocolHandler
["https-openssl-nio-10.2.251.132-443"]
2023-07-10T20:52:06.346Z INFO Server startup in [28980] milliseconds
2023-07-10T21:35:40.487Z INFO Pausing ProtocolHandler
["https-openssl-nio-10.2.251.132-443"]
2023-07-10T21:35:40.495Z INFO Stopping service [Catalina]
2023-07-10T21:35:40.910Z INFO Destroyed pool :
|default|lo|-2023-07-10T20-51-53.099285500Z
Good point: Tomcat will complain about this stuff on shutdown even if the
warnings are irrelevant because the application-stop leak-checks are run
whether Tomcat is going to stop or not.
So operationally, these warnings may be irrelevant.
(IMHO you should still fix the application!)
Another thing I noticed was the difference in allocated Java heap. It
appears that first it sets min=max=1GB and after that it's setting max
heap to 256MB. That could be the problem. JVM might be crashing
because of heap shortage.
10-Jul-2023 16:51:35.481 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dconfig.url=D:\ords222
10-Jul-2023 16:51:35.481 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Xms1024M
10-Jul-2023 16:51:35.481 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Xmx1024M
10-Jul-2023 16:51:35.486 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: exit
10-Jul-2023 16:51:35.486 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: abort
10-Jul-2023 16:51:35.486 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Xms128m
10-Jul-2023 16:51:35.486 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Xmx256m
Oh, I hadn't noticed that, either.
Dropping your heap to 25% of original values could certainly cause problems :)
I don't see a "crash" in the log, so it's hard to tell what exactly is being
reported, here.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
