Olaf and Sevendu, Thank you for your replies. Correct, I sincerely doubt this is a Tomcat class loading bug.
I am using Tomcat’s normal class loader (webapp/WAR) to load the classes into memory, and it is a single class loader. I am going to periodically run: jcmd <pid> GC.class_stats I am only having the issue in general operation and not on a redeploy, and I have to restart the server daily. I will find out more details as to how these classes are loaded into memory. Here is what is going on… I have a 16GB Linux instance and one Apache Tomcat 9.0.78 instance running on it. It is running JDK 1.8.0_371-b25. I have min and max JVM heap setting at 8GB. JVM arguments are: -XX:CICompilerCount=3 -XX:ConcGCThreads=1 -XX:+DisableExplicitGC -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=3145728 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=8589934592 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=8589934592 -XX:MaxNewSize=5150605312 -XX:MinHeapDeltaBytes=4194304 -XX:NativeMemoryTracking=detail -XX:NumberOfGCLogFiles=25 -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation The MaxMetaspaceSize is not set, so this means it is unlimited. After the server comes up and after a short period of time I get a native out of memory condition: # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory. # Possible reasons: # The system is out of physical RAM or swap space # The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (os_linux.cpp:2798), pid=191803, tid=0x000014fff94b7700 # # JRE version: (8.0_371) (build ) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.371-b25 mixed mode linux-amd64 compressed oops) # Core dump written. Default location: /hosting/core or core.191803 # Regards, William Crowell From: Olaf Kock <tom...@olafkock.de> Date: Tuesday, November 7, 2023 at 4:25 AM To: users@tomcat.apache.org <users@tomcat.apache.org> Subject: Re: Are there any known class loader leaks in Tomcat 9? On 06.11.23 18:55, William Crowell wrote: > Good afternoon, > > I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am > loading some very large Java classes into Metaspace. I know this is not good > practice, but I inherited this library. These classes have business rules > and are doing some basic primitive and array manipulations, and I am running > out of native memory. I don’t think there are any JNI calls in this code > base. > > Are there anything existing issues with the Tomcat 9 class loader? I doubt > there are but wanted to check. Hi William, when you say "I am loading some very large Java classes into Metaspace. I know this is not good practice" it does not sound like this inherited library is using tomcat's standard way of classloading - please clarify. Also, nothing in that statement points to anything where a leak on tomcat's side would make a difference: Are you experiencing problems after reloading a webapp, or just in general operation? You might simply need to assign more metaspace to the VM for this library to operate properly. So, this boils down to: * How does this library load the classes into memory? * Do you redeploy? * What's your metaspace (and general memory) setting, and what are the conditions under which you run out of memory? * Do you see any log message that hint at classes that won't be garbage collected because of stale references? Tomcat issues those warnings, in case you (for example) start your own background threads that keep holding the memory allocated to them. Also: I second your doubt about Tomcat's classloader being the cause for your problems - unless proven otherwise. Olaf CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
