-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Mark,
On 4/24/20 17:46, Mark Boon wrote: > Thanks Chris for taking the time. > > As you point out, from the threads I can tell we're not using ARP > as the names al all starting with "jsse". AFAI could find out BouncyCastle is a pure Java implementation, so that also can't be the cause. > > Someone suggested PAMLibrary may be the culprit. So I started a thread that makes continuous auth calls to the PAM library. Now there does seems to be an indication memory is leaking very, very slowly. It seems to be roughly in line with the number of auth failures. It looks like PAM throttles auth failures though, hence it's taking such a long time for the evidence to mount. > > So nothing to see here for this group. Just wanted to give a heads > up. Actually, this is great feedback, even if it means there is no action to take by the Tomcat team. 312 bytes leaked per allocation is enough to be dangerous, but not such a huge problem to make it easy to discover . Identifying that it is an authentication library that isn't commonly used with Java (or maybe it is?!) at least puts it into the archives of this list in case someone else is having a similar problem. Thanks, - -chris > On 4/6/20, 12:12 PM, "Christopher Schultz" <ch...@christopherschultz.net> wrote: > > Mark, > > On 4/3/20 21:48, Mark Boon wrote: >> For the past few months we’ve been trying to trace what looks >> like gradual memory creep. After some long-running experiments it >> seems due to memory leaking when jni_invoke_static(JNIEnv_*, >> JavaValue*, _jobject*, JNICallType, _jmethodID*, >> JNI_ArgumentPusher*, Thread*) is invoked. Somewhere. > >> My environment is Tomcat running a proxy webapp. It does TLS >> termination, authentication and then forwards the call to local >> services. It doesn’t do much else, it’s a relatively small >> application. > >> Some (possibly relevant) versions and config parameters: Tomcat >> 8.5 Java 8u241 (Oracle) Heap size = 360Mb MAX_ALLOC_ARENA=2 >> MALLOC_TRIM_THRESHOLD_=250048 jdk.nio.maxCachedBufferSize=25600 > >> We couldn’t find any proof of memory leaking on the Java side. >> When we turn on NativeMemoryTracking=detail and we take a >> snapshot shortly after starting, we see (just one block shown): > >> [0x000003530e462f9a] >> JNIHandleBlock::allocate_block(Thread*)+0xaa [0x000003530e3f759a] >> JavaCallWrapper::JavaCallWrapper(methodHandle, Handle, >> JavaValue*, Thread*)+0x6a [0x000003530e3fa000] >> JavaCalls::call_helper(JavaValue*, methodHandle*, >> JavaCallArguments*, Thread*)+0x8f0 [0x000003530e4454a1] >> jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, >> _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] >> [clone .constprop.117]+0x1e1 (malloc=33783KB type=Internal >> #110876) > >> Then we run it under heavy load for a few weeks and take another >> snapshot: > >> [0x000003530e462f9a] >> JNIHandleBlock::allocate_block(Thread*)+0xaa [0x000003530e3f759a] >> JavaCallWrapper::JavaCallWrapper(methodHandle, Handle, >> JavaValue*, Thread*)+0x6a [0x000003530e3fa000] >> JavaCalls::call_helper(JavaValue*, methodHandle*, >> JavaCallArguments*, Thread*)+0x8f0 [0x000003530e4454a1] >> jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, >> _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] >> [clone .constprop.117]+0x1e1 (malloc=726749KB type=Internal >> #2385226) > >> While other blocks also show some variation, none show growth >> like this one. When I do some math on the number (726749KB - >> 33783KB) / (2385226 – 110876) it comes down to a pretty even 312 >> bytes per allocation. And we leaked just under 700Mb. While not >> immediately problematic, this does not bode well for our >> customers who run this service for months. > >> I’d like to avoid telling them they need to restart this service >> every two weeks to reclaim memory. Has anyone seen something >> like this? Any way it could be avoided? > > That was some very good sleuthing on your part. 312 bytes per > allocation will indeed be very difficult to detect unless you are > really looking hard for it. > > On 4/4/20 13:02, Mark Boon wrote: >> The connector of the webapp uses Http11NioProtocol. My >> understanding is it uses direct-byte-buffers backed by native >> memory for the Nio channels. I don't know for sure if that gets >> allocated through a JNI call, but that was my assumption. > > This will definitely use Tomcat's NIO protocol which doesn't use > the APR connector. However, you still might be using tcnative to > get the crypto engine. Can you confirm the thread-naming convention > of your request-processing threads? They will tell you if JSSE or > OpenSSL (tcnative) is being used. > > A few data points: > > * No Tomcat code directly invokes jni_invoke_static(), but it might > do so indirectly through a variety of means. > > * NIO does use buffers, but those buffers tend to be (a) fairly > large -- on the order of kilobytes -- and (b) re-used for the life > of the request-processor thread. > > It is very possible that there is a very small leak in Tomcat's > handling of NIO buffers. I think it's equally likely that there is > a bug in the JVM itself. > > Are you able to try different JVM versions in your test? I would > recommend major-version changes, here. I thought I read somewhere > that Oracle re-wrote the implementation of the NIO API in a > somewhat recent Java release (Java 9?), but I can't seem to find > that reference, now. > > Are you able to try: > > - Java 8 - Java 9/10/11/12 - Java 13 > > -chris > > PS This bug report may be relevant: > https://bugs.openjdk.java.net/browse/JDK-8190395 > > The bug report says it's closed/incomplete, but they do mention a > 312-byte leak with certain invocations. > > - --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For > additional commands, e-mail: users-h...@tomcat.apache.org > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl6lp2cACgkQHPApP6U8 pFip2g/7BNx/zSvRTKBP2NS7OoZGEcOG69S/WOA6AJQWWoieFPbDfXJt80haRsof 8krc0jCsdBcBl1v2XBHwYVrj9hpSribmc6TUqexAVlMI+T+2T33h6YPe5OApESjg tpAlDaP4fyGSu9rNdJ2kTakB+v9zB1FaIzyk0c5IiN5YuzngP1dwcJ2xLXr2MaM4 Mng4Z7CtEMzv5p9KAzkYORFCKd1eXFWDHmuLlSKKtjR+c1zcQ7R8jwB87ZUl8QYC 4W+83JPk/0goVNzBQbOBj/EaWQd+sYb5nOy5VQQlj7IVs9pJWmhAhD1TXncj5ZTT sfaaN0mvsdIW7axvPHd1yH0MdFQm4iN+6KW/52E5Mpo8BzT1I9s2Xn4Bu6DPfnXc 92PFT3F6cbQQwEvRTNTPN1uQ+paZM4fPztckHKrpFPKKXUWYWZtYjA7hNR0kRj1R sk+R882VXuvzOBB2KavqpNDoW5LHvbQ1I/1qpwJFIGwZ0V2Q2YZ3KRCKKLQB0XLV 8Qytg1Iyp57fQRE/uT2ZCTI3X+aOnBhKMi8AyVaEXa132tO2M3yjIE/TsYJj6x3M GRkTsUbtJxaTsa+0+49uqMTxOcFfh2FlHrLr43QZY+gKfzXa/hiDGVW0kC3nj0As UhvFgEtEbXifFoR2uvLzz25JmvUB7jXe9CMs92Jm2yu0NlJK+es= =f3ra -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
