Thanks for the input. I will forward the email to our developers to look at the heap size settings being different. We have a Windows service that is used to start/stop Tomcat. When this happens we find that the Windows service is no longer running.
Thanks, James Boggs -----Original Message----- From: Christopher Schultz <ch...@christopherschultz.net> Sent: Wednesday, July 12, 2023 3:32 PM To: users@tomcat.apache.org Subject: Re: Tomcat 9.0.76 Memory leak with Java 17 Suvendu, On 7/12/23 07:11, Suvendu Sekhar Mondal wrote: > On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz > <ch...@christopherschultz.net> wrote: >> >> James, >> >> On 7/11/23 10:21, James Boggs wrote: >>> We had a stable SSL enabled website with Apache Tomcat 9.0.73 on >>> Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL. >>> >>> We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and >>> to ORDS 22.1, then used Java 17 to create a new Java Keystore and a >>> new SSL csr file, and imported a new SSL certificate from the CA >>> into the new keystore. >>> >>> The website works but after logging in there are memory leak >>> warnings and the Tomcat service crashes within just a couple of minutes. >>> >>> We even upgraded to 9.0.76 and the issue persists. >>> >>> Below is an excerpt from the stderr log file. >>> >>> I have been unable to find any recent threads on this, any help is >>> appreciated. Is any other information needed? >> >> I think you have included all necessary information. I'm chopping-out >> the irrrelevant bits: >> >>> 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd] >>> registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to >>> unregister it when the web application was stopped. To prevent a >>> memory leak, the JDBC Driver has been forcibly unregistered. >> >> This is actually "okay" in that Tomcat has detected a global JDBC >> driver registration performed by the application, and is fixing the >> problem for you. The application, however, is making a mistake by not >> de-registering that JDBC driver. Your options are to move the driver >> library from your application into Tomcat's lib/ directory, fix the >> application to de-register the driver when it shuts down, or just ignore the >> warning. >> But you should fix the application. >> >>> 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd] >>> appears to have started a thread named >>> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] >>> but has failed to stop it. This is very likely to create a memory leak. >>> Stack trace of thread: >> >> There are multiple instances of this same message and THIS is your >> problem. The problem is what the error message says: your application >> has started a Thread and never stopped it. The "memory leak" comes >> from the fact that the Thread has inherited the web application's >> ClassLoader and the web application is being re-loaded. When that >> happens, Tomcat discards the ClassLoader which usually means the GC >> will clean up after it at some point in the future. But that Thread >> is still running and will keep the ClassLoader in memory, likely forever. >> >> You have a few options: >> >> 1. Fix the application. The application needs to shut-down any >> threads is starts during its operation, preferrably in a >> ServletContextListener's destroy method or similar. >> >> 2. Don't hot-reload your application. Instead, shut-down the JVM and >> re-start it. Ovbviously, this may have availability implications, but >> then again so does running out of memory and having to bounce the >> JVM, anyway. >> > > I was checking the stacks and saw this: They might not be doing > "hot-deploy" of the app. AFAIK those messages come once someone stops > Tomcat. > >> 2023-07-10T20:52:06.292Z INFO Starting ProtocolHandler >> ["https-openssl-nio-10.2.251.132-443"] >> 2023-07-10T20:52:06.346Z INFO Server startup in [28980] milliseconds >> 2023-07-10T21:35:40.487Z INFO Pausing ProtocolHandler >> ["https-openssl-nio-10.2.251.132-443"] >> 2023-07-10T21:35:40.495Z INFO Stopping service [Catalina] >> 2023-07-10T21:35:40.910Z INFO Destroyed pool : >> |default|lo|-2023-07-10T20-51-53.099285500Z Good point: Tomcat will complain about this stuff on shutdown even if the warnings are irrelevant because the application-stop leak-checks are run whether Tomcat is going to stop or not. So operationally, these warnings may be irrelevant. (IMHO you should still fix the application!) > Another thing I noticed was the difference in allocated Java heap. It > appears that first it sets min=max=1GB and after that it's setting max > heap to 256MB. That could be the problem. JVM might be crashing > because of heap shortage. > >> 10-Jul-2023 16:51:35.481 INFO [main] >> org.apache.catalina.startup.VersionLoggerListener.log Command line >> argument: -Dconfig.url=D:\ords222 >> 10-Jul-2023 16:51:35.481 INFO [main] >> org.apache.catalina.startup.VersionLoggerListener.log Command line >> argument: -Xms1024M >> 10-Jul-2023 16:51:35.481 INFO [main] >> org.apache.catalina.startup.VersionLoggerListener.log Command line >> argument: -Xmx1024M > >> 10-Jul-2023 16:51:35.486 INFO [main] >> org.apache.catalina.startup.VersionLoggerListener.log Command line >> argument: exit >> 10-Jul-2023 16:51:35.486 INFO [main] >> org.apache.catalina.startup.VersionLoggerListener.log Command line >> argument: abort >> 10-Jul-2023 16:51:35.486 INFO [main] >> org.apache.catalina.startup.VersionLoggerListener.log Command line >> argument: -Xms128m >> 10-Jul-2023 16:51:35.486 INFO [main] >> org.apache.catalina.startup.VersionLoggerListener.log Command line >> argument: -Xmx256m Oh, I hadn't noticed that, either. Dropping your heap to 25% of original values could certainly cause problems :) I don't see a "crash" in the log, so it's hard to tell what exactly is being reported, here. -chris --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
