Mark Thanks again for taking the time to assist with the OutOfMemoryError. BLUF, it looks like Log4J2 is the culprit. Will you please check my work below to see if I have interpreted correctly?
-----Original Message----- From: Mark Thomas [mailto:ma...@apache.org] Sent: Wednesday, July 20, 2016 12:17 PM To: Tomcat Users List Subject: Re: OutOfMemoryError: PermGen space > On 20/07/2016 17:31, Berneburg, Cris wrote: > > Mark > > > > Thanks again for taking the time to educate me and answer my > > questions. My lengthy replies below. To summarize, our app does not > > seem to commit any egregious memory leak offenses, from what I can > > tell so far. I plan to heap dump an older version of the app that > > does not use third-party libraries for comparison purposes. Using the Find Leaks button on the Tomcat Manager page on old app versions to trigger full garbage collection revealed that the memory leak started to happen in the release when Log4J2 was added to the app. It did not start happening in the prior release when Mybatis was added. > This should help: > http://markmail.org/message/fcbvwapt6afyndxn > 1. Find an app that you can't reload without OOME > 2. Get a profiler [...] > 3. Reload you app once > 4. Use the profiler to look for instances of WebappClassLoader > 5. Look for the one with the started attribute == false > 6. Trace the GC roots for this instance Used the Java Visual VM to pull a heap dump after the app restart and GC. Used Eclipse Memory Analyzer to analyze the heap dumps. Found the WebappClassLoader with started == false and used Path to GC roots: <classloader> org.apache.logging.log4j.core.jmx.LoggerContextAdminMBean ... com.sun.jmx.mbeanserver.StandardMBeanIntrospector <classloader> org.apache.logging.log4j.core.jmx.LoggerContextAdmin ... com.sun.jmx.mbeanserver.StandardMBeanIntrospector <classloader> org.apache.logging.log4j.core.jmx.StatusLoggerAdmin ... com.sun.jmx.mbeanserver.StandardMBeanIntrospector referent java.util.WeakHashMap$Entry ... java.lang.reflect.Proxy <classloader> $Proxy3 ... java.lang.reflect.Proxy referent java.util.WeakHashMap$Entry ... org.apache.juli.ClassLoaderLogManager ... (many java.util.logging.*) I don't see anything pointing back to our code. With no previous experience with heap analysis on my part, it looks to me to be due to Log4J2. Changing the args for the call to LogManager.getLogger from Class<T> clazz to none made no appreciable difference. (Did I somehow set up Log4J2 incorrectly to get it to misbehave?) Not sure what else I can do. Report it to the Log4J2 dev group perhaps? > One thing worth noting. Yourkit offers two types of heap dumps. > You don't want YourKit's own. There is a JVM bug that prevents > some GC roots from being shown in that format and if your leak > traces back to one of those you can end up scratching your head > for days. > > Mark Is there a likely chance that the bug interfered with my usage of Java Visual VM and Eclipse Memory Analyzer and thus prevented me from determining the real problem source? -- Cris Berneburg, Lead Software Engineer, CACI --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
