On Sat, Apr 4, 2020, 12:02 Mark Boon <mb...@vmware.com.invalid> wrote:
> I don't have 'proof' Tomcat is to blame. Hence the question-mark. All I > have managed is narrow it down to this NMT data, which is not very > informative. I hoped anyone could give me an idea how or where to > investigate further. Or if someone had run into this before. > > The connector of the webapp uses Http11NioProtocol. My understanding is it > uses direct-byte-buffers backed by native memory for the Nio channels. I > don't know for sure if that gets allocated through a JNI call, but that was > my assumption. > > I did not consider trying Mission Control or jvisualvm. Isn't Mission > Control for embedded Java? And AFAIK, jvisualvm is for profiling Java > memory usage and underneath uses tools like jmap, jstat and jcmd. Through > GC logs and jmap heap-dumps I can confidently say there's no memory leak on > the Java side. The NMT data shown comes from jcmd. No type grows beyond > control and full GC always returns to the same baseline for the heap. > Anyway, the Java heap is only 360Mb and this memory-block created by > jni_invoke_static has grown to 700Mb by itself. And I see no out-of-memory > messages. The only hint of this happening is that the RES memory of the > Tomcat process keeps growing over time, as shown by 'top'. And it seems GC > is getting slower over time, but the customers haven't noticed it yet. > (This is after we switched to ParallelGC. We did see considerable slow-down > when using G1GC in the ref-processing, but we couldn't figure out why. It > would slow to a crawl before the memory leak became obvious.) > > Anyway, I was mostly fishing for hints or tips that could help me figure > this out or avoid it. > > The application is simple to the point I'm hard-pressed to think of any > other part making JNI calls. The only library I can think of using JNI is > BouncyCastle doing the SSL encryption/decryption, so maybe I'll switch my > focus there. > Something else to consider - we should keep in mind that a JVM is loaded for the native code, but won't be obvious in a process table : )
