-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Mark,
On 4/3/20 21:48, Mark Boon wrote: > For the past few months we’ve been trying to trace what looks like > gradual memory creep. After some long-running experiments it seems > due to memory leaking when jni_invoke_static(JNIEnv_*, JavaValue*, > _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) > is invoked. Somewhere. > > My environment is Tomcat running a proxy webapp. It does TLS > termination, authentication and then forwards the call to local > services. It doesn’t do much else, it’s a relatively small > application. > > Some (possibly relevant) versions and config parameters: Tomcat > 8.5 Java 8u241 (Oracle) Heap size = 360Mb MAX_ALLOC_ARENA=2 > MALLOC_TRIM_THRESHOLD_=250048 jdk.nio.maxCachedBufferSize=25600 > > We couldn’t find any proof of memory leaking on the Java side. When > we turn on NativeMemoryTracking=detail and we take a snapshot > shortly after starting, we see (just one block shown): > > [0x000003530e462f9a] JNIHandleBlock::allocate_block(Thread*)+0xaa > [0x000003530e3f759a] JavaCallWrapper::JavaCallWrapper(methodHandle, > Handle, JavaValue*, Thread*)+0x6a [0x000003530e3fa000] > JavaCalls::call_helper(JavaValue*, methodHandle*, > JavaCallArguments*, Thread*)+0x8f0 [0x000003530e4454a1] > jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, > _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone > .constprop.117]+0x1e1 (malloc=33783KB type=Internal #110876) > > Then we run it under heavy load for a few weeks and take another > snapshot: > > [0x000003530e462f9a] JNIHandleBlock::allocate_block(Thread*)+0xaa > [0x000003530e3f759a] JavaCallWrapper::JavaCallWrapper(methodHandle, > Handle, JavaValue*, Thread*)+0x6a [0x000003530e3fa000] > JavaCalls::call_helper(JavaValue*, methodHandle*, > JavaCallArguments*, Thread*)+0x8f0 [0x000003530e4454a1] > jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, > _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone > .constprop.117]+0x1e1 (malloc=726749KB type=Internal #2385226) > > While other blocks also show some variation, none show growth like > this one. When I do some math on the number (726749KB - 33783KB) / > (2385226 – 110876) it comes down to a pretty even 312 bytes per > allocation. And we leaked just under 700Mb. While not immediately > problematic, this does not bode well for our customers who run this > service for months. > > I’d like to avoid telling them they need to restart this service > every two weeks to reclaim memory. Has anyone seen something like > this? Any way it could be avoided? That was some very good sleuthing on your part. 312 bytes per allocation will indeed be very difficult to detect unless you are really looking hard for it. On 4/4/20 13:02, Mark Boon wrote: > The connector of the webapp uses Http11NioProtocol. My > understanding is it uses direct-byte-buffers backed by native > memory for the Nio channels. I don't know for sure if that gets > allocated through a JNI call, but that was my assumption. This will definitely use Tomcat's NIO protocol which doesn't use the APR connector. However, you still might be using tcnative to get the crypto engine. Can you confirm the thread-naming convention of your request-processing threads? They will tell you if JSSE or OpenSSL (tcnative) is being used. A few data points: * No Tomcat code directly invokes jni_invoke_static(), but it might do so indirectly through a variety of means. * NIO does use buffers, but those buffers tend to be (a) fairly large - -- on the order of kilobytes -- and (b) re-used for the life of the request-processor thread. It is very possible that there is a very small leak in Tomcat's handling of NIO buffers. I think it's equally likely that there is a bug in the JVM itself. Are you able to try different JVM versions in your test? I would recommend major-version changes, here. I thought I read somewhere that Oracle re-wrote the implementation of the NIO API in a somewhat recent Java release (Java 9?), but I can't seem to find that reference, now. Are you able to try: - - Java 8 - - Java 9/10/11/12 - - Java 13 - -chris PS This bug report may be relevant: https://bugs.openjdk.java.net/browse/JDK-8190395 The bug report says it's closed/incomplete, but they do mention a 312-byte leak with certain invocations. -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl6Lfx8ACgkQHPApP6U8 pFiGEQ/7BweMhSjALeMhEoAURv6NarT33Nb1ydHynOmDXnSXbczO+B6MC1j9QjHD 2sdYsPRLpi8holt2pl3dLxrsWE4gkK27e+2hwNN7568P/9S03m9VzpdciuwBCwfu 0acFtRs8iWcJO0jI29R438lt6w1CY6QGV2rasZKhkhaoBA8K9mSb0J02KUhDlpbn oljBKA2k1oeqEDJpJoejsX7Zwgaf2aM96VZNds8Atly1WZhqj0nENwtU4yaIxel5 HyHrjOLsHMLPZeTDx/5pFs45qTGfFos1YF1lf99EDdKuX9qv+X+Dr2vr0RZW2iw8 5Oxh0EqAjMko9ysvjo2N7cnKK5fLNvAHRAGBIqDlAtcWF51vddeMASHjKYmM+/ha pvN3/Dff65QXK74fgIk10Yqro1REudYcXTXT2+9WdycS06HOORndQaMs3l0qqaLG 057oHe9XuZs2th5imcS1D8zwaOmXo6jQmDYtGNEYk7pG8pfGfnHgnmVYCtGs067V Rus6hMQAiUnEDgWVB/tYHZVCvzpR3nPVY5eckwComaSaSHD4p1nosRe7ILgaGJOu fIxQGZ6PzgAAQZHgdqwkPeNLygZ3I46Nnliazdkejimyir6NLje8YW83LsWdlI5l FlpdEkb92TA2TljDadkdHuUJGw+nFZmDla+AwZ4WPgI18PleBJs= =PPJb -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
