Chris, Yes it is unintentional. Actually once we start it with the Windows service, and run through a few reports on the website, it stops in just ba few minutes. We will look at the java heap size settings.
Regards, James Boggs -----Original Message----- From: Christopher Schultz <ch...@christopherschultz.net> Sent: Wednesday, July 12, 2023 3:55 PM To: users@tomcat.apache.org Subject: Re: Tomcat 9.0.76 Memory leak with Java 17 James, On 7/12/23 15:41, James Boggs wrote: > Thanks for the input. I will forward the email to our developers to > look at the heap size settings being different. > > We have a Windows service that is used to start/stop Tomcat. When this > happens we find that the Windows service is no longer running. Depending upon your settings, you may have more than one log file to look at. Look at everything in CATALINA_BASE/logs the next time the service goes down to see if there is a clue in any of them. You could also look for hs_pid[process-id] files (or the Windows equivalent, if it's different) that are usually dumped when a JVM crash occurs. Each of your logs seems to show that Tomcat is being shut-down in an orderly way, though. Are you sure it's crashing and not being intentionally stopped? -chris > -----Original Message----- > From: Christopher Schultz <ch...@christopherschultz.net> > Sent: Wednesday, July 12, 2023 3:32 PM > To: users@tomcat.apache.org > Subject: Re: Tomcat 9.0.76 Memory leak with Java 17 > > Suvendu, > > On 7/12/23 07:11, Suvendu Sekhar Mondal wrote: >> On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz >> <ch...@christopherschultz.net> wrote: >>> >>> James, >>> >>> On 7/11/23 10:21, James Boggs wrote: >>>> We had a stable SSL enabled website with Apache Tomcat 9.0.73 on >>>> Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL. >>>> >>>> We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 >>>> and to ORDS 22.1, then used Java 17 to create a new Java Keystore >>>> and a new SSL csr file, and imported a new SSL certificate from the >>>> CA into the new keystore. >>>> >>>> The website works but after logging in there are memory leak >>>> warnings and the Tomcat service crashes within just a couple of minutes. >>>> >>>> We even upgraded to 9.0.76 and the issue persists. >>>> >>>> Below is an excerpt from the stderr log file. >>>> >>>> I have been unable to find any recent threads on this, any help is >>>> appreciated. Is any other information needed? >>> >>> I think you have included all necessary information. I'm >>> chopping-out the irrrelevant bits: >>> >>>> 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd] >>>> registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to >>>> unregister it when the web application was stopped. To prevent a >>>> memory leak, the JDBC Driver has been forcibly unregistered. >>> >>> This is actually "okay" in that Tomcat has detected a global JDBC >>> driver registration performed by the application, and is fixing the >>> problem for you. The application, however, is making a mistake by >>> not de-registering that JDBC driver. Your options are to move the >>> driver library from your application into Tomcat's lib/ directory, >>> fix the application to de-register the driver when it shuts down, or just >>> ignore the warning. >>> But you should fix the application. >>> >>>> 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd] >>>> appears to have started a thread named >>>> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] >>>> but has failed to stop it. This is very likely to create a memory leak. >>>> Stack trace of thread: >>> >>> There are multiple instances of this same message and THIS is your >>> problem. The problem is what the error message says: your >>> application has started a Thread and never stopped it. The "memory >>> leak" comes from the fact that the Thread has inherited the web >>> application's ClassLoader and the web application is being >>> re-loaded. When that happens, Tomcat discards the ClassLoader which >>> usually means the GC will clean up after it at some point in the >>> future. But that Thread is still running and will keep the ClassLoader in >>> memory, likely forever. >>> >>> You have a few options: >>> >>> 1. Fix the application. The application needs to shut-down any >>> threads is starts during its operation, preferrably in a >>> ServletContextListener's destroy method or similar. >>> >>> 2. Don't hot-reload your application. Instead, shut-down the JVM and >>> re-start it. Ovbviously, this may have availability implications, >>> but then again so does running out of memory and having to bounce >>> the JVM, anyway. >>> >> >> I was checking the stacks and saw this: They might not be doing >> "hot-deploy" of the app. AFAIK those messages come once someone stops >> Tomcat. >> >>> 2023-07-10T20:52:06.292Z INFO Starting ProtocolHandler >>> ["https-openssl-nio-10.2.251.132-443"] >>> 2023-07-10T20:52:06.346Z INFO Server startup in [28980] milliseconds >>> 2023-07-10T21:35:40.487Z INFO Pausing ProtocolHandler >>> ["https-openssl-nio-10.2.251.132-443"] >>> 2023-07-10T21:35:40.495Z INFO Stopping service [Catalina] >>> 2023-07-10T21:35:40.910Z INFO Destroyed pool : >>> |default|lo|-2023-07-10T20-51-53.099285500Z > > Good point: Tomcat will complain about this stuff on shutdown even if the > warnings are irrelevant because the application-stop leak-checks are run > whether Tomcat is going to stop or not. > > So operationally, these warnings may be irrelevant. > > (IMHO you should still fix the application!) > >> Another thing I noticed was the difference in allocated Java heap. It >> appears that first it sets min=max=1GB and after that it's setting >> max heap to 256MB. That could be the problem. JVM might be crashing >> because of heap shortage. >> >>> 10-Jul-2023 16:51:35.481 INFO [main] >>> org.apache.catalina.startup.VersionLoggerListener.log Command line >>> argument: -Dconfig.url=D:\ords222 >>> 10-Jul-2023 16:51:35.481 INFO [main] >>> org.apache.catalina.startup.VersionLoggerListener.log Command line >>> argument: -Xms1024M >>> 10-Jul-2023 16:51:35.481 INFO [main] >>> org.apache.catalina.startup.VersionLoggerListener.log Command line >>> argument: -Xmx1024M >> >>> 10-Jul-2023 16:51:35.486 INFO [main] >>> org.apache.catalina.startup.VersionLoggerListener.log Command line >>> argument: exit >>> 10-Jul-2023 16:51:35.486 INFO [main] >>> org.apache.catalina.startup.VersionLoggerListener.log Command line >>> argument: abort >>> 10-Jul-2023 16:51:35.486 INFO [main] >>> org.apache.catalina.startup.VersionLoggerListener.log Command line >>> argument: -Xms128m >>> 10-Jul-2023 16:51:35.486 INFO [main] >>> org.apache.catalina.startup.VersionLoggerListener.log Command line >>> argument: -Xmx256m > > Oh, I hadn't noticed that, either. > > Dropping your heap to 25% of original values could certainly cause > problems :) > > I don't see a "crash" in the log, so it's hard to tell what exactly is being > reported, here. > > -chris > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
