Hi Chris, On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz <ch...@christopherschultz.net> wrote: > > James, > > On 7/11/23 10:21, James Boggs wrote: > > We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows > > Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL. > > > > We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to > > ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL > > csr file, and imported a new SSL certificate from the CA into the new > > keystore. > > > > The website works but after logging in there are memory leak warnings > > and the Tomcat service crashes within just a couple of minutes. > > > > We even upgraded to 9.0.76 and the issue persists. > > > > Below is an excerpt from the stderr log file. > > > > I have been unable to find any recent threads on this, any help is > > appreciated. Is any other information needed? > > I think you have included all necessary information. I'm chopping-out > the irrrelevant bits: > > > 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd] > > registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to > > unregister it when the web application was stopped. To prevent a memory > > leak, the JDBC Driver has been forcibly unregistered. > > This is actually "okay" in that Tomcat has detected a global JDBC driver > registration performed by the application, and is fixing the problem for > you. The application, however, is making a mistake by not de-registering > that JDBC driver. Your options are to move the driver library from your > application into Tomcat's lib/ directory, fix the application to > de-register the driver when it shuts down, or just ignore the warning. > But you should fix the application. > > > 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd] > > appears to have started a thread named > > [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] > > but has failed to stop it. This is very likely to create a memory leak. > > Stack trace of thread: > > There are multiple instances of this same message and THIS is your > problem. The problem is what the error message says: your application > has started a Thread and never stopped it. The "memory leak" comes from > the fact that the Thread has inherited the web application's ClassLoader > and the web application is being re-loaded. When that happens, Tomcat > discards the ClassLoader which usually means the GC will clean up after > it at some point in the future. But that Thread is still running and > will keep the ClassLoader in memory, likely forever. > > You have a few options: > > 1. Fix the application. The application needs to shut-down any threads > is starts during its operation, preferrably in a > ServletContextListener's destroy method or similar. > > 2. Don't hot-reload your application. Instead, shut-down the JVM and > re-start it. Ovbviously, this may have availability implications, but > then again so does running out of memory and having to bounce the JVM, > anyway. >
I was checking the stacks and saw this: They might not be doing "hot-deploy" of the app. AFAIK those messages come once someone stops Tomcat. > 2023-07-10T20:52:06.292Z INFO Starting ProtocolHandler > ["https-openssl-nio-10.2.251.132-443"] > 2023-07-10T20:52:06.346Z INFO Server startup in [28980] milliseconds > 2023-07-10T21:35:40.487Z INFO Pausing ProtocolHandler > ["https-openssl-nio-10.2.251.132-443"] > 2023-07-10T21:35:40.495Z INFO Stopping service [Catalina] > 2023-07-10T21:35:40.910Z INFO Destroyed pool : > |default|lo|-2023-07-10T20-51-53.099285500Z Another thing I noticed was the difference in allocated Java heap. It appears that first it sets min=max=1GB and after that it's setting max heap to 256MB. That could be the problem. JVM might be crashing because of heap shortage. > 10-Jul-2023 16:51:35.481 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Command line argument: > -Dconfig.url=D:\ords222 > 10-Jul-2023 16:51:35.481 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Command line argument: > -Xms1024M > 10-Jul-2023 16:51:35.481 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Command line argument: > -Xmx1024M > 10-Jul-2023 16:51:35.486 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Command line argument: > exit > 10-Jul-2023 16:51:35.486 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Command line argument: > abort > 10-Jul-2023 16:51:35.486 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Command line argument: > -Xms128m > 10-Jul-2023 16:51:35.486 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Command line argument: > -Xmx256m > > 2023-07-10T21:35:40.944Z SEVERE The web application [rplans-vpd] > > created a ThreadLocal with key of type > > [java.lang.ThreadLocal.SuppliedThreadLocal] (value > > [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of > > type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to > > remove it when the web application was stopped. Threads are going to be > > renewed over time to try and avoid a probable memory leak. > > This is actually "okay" in that Tomcat has detected your application's > ThreadLocal variables (objects bound to one or more Threads which are > owned by the application) which are not being cleaned-up by the > application, and it's fixing the problem for you by, over time, killing > each of those Threads and replacing them in the Thread pool for you. > Your options are to fix the application or to ignore the warning. But > you should fix the application. > > It appears that your upgrade of ORDS has introduced a lot of stuff that > doesn't play well with hot-reloading of the application. I'm assuming > that you aren't responsible for maintaining ORDS... Oracle is. > > You should report all of the previous issues to Oracle against their > ORDS version 22.1 and ask them to fix them. It's why you write those > big, fat checks in the first place ;) > > -chris > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
