Thanks Chris for taking the time.
As you point out, from the threads I can tell we're not using ARP as the names
al all starting with "jsse". AFAI could find out BouncyCastle is a pure Java
implementation, so that also can't be the cause.
Someone suggested PAMLibrary may be the culprit. So I started a thread that
makes continuous auth calls to the PAM library. Now there does seems to be an
indication memory is leaking very, very slowly. It seems to be roughly in line
with the number of auth failures. It looks like PAM throttles auth failures
though, hence it's taking such a long time for the evidence to mount.
So nothing to see here for this group. Just wanted to give a heads up.
Mark
On 4/6/20, 12:12 PM, "Christopher Schultz" <ch...@christopherschultz.net>
wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mark,
On 4/3/20 21:48, Mark Boon wrote:
> For the past few months we’ve been trying to trace what looks like
> gradual memory creep. After some long-running experiments it seems
> due to memory leaking when jni_invoke_static(JNIEnv_*, JavaValue*,
> _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)
> is invoked. Somewhere.
>
> My environment is Tomcat running a proxy webapp. It does TLS
> termination, authentication and then forwards the call to local
> services. It doesn’t do much else, it’s a relatively small
> application.
>
> Some (possibly relevant) versions and config parameters: Tomcat
> 8.5 Java 8u241 (Oracle) Heap size = 360Mb MAX_ALLOC_ARENA=2
> MALLOC_TRIM_THRESHOLD_=250048 jdk.nio.maxCachedBufferSize=25600
>
> We couldn’t find any proof of memory leaking on the Java side. When
> we turn on NativeMemoryTracking=detail and we take a snapshot
> shortly after starting, we see (just one block shown):
>
> [0x000003530e462f9a] JNIHandleBlock::allocate_block(Thread*)+0xaa
> [0x000003530e3f759a] JavaCallWrapper::JavaCallWrapper(methodHandle,
> Handle, JavaValue*, Thread*)+0x6a [0x000003530e3fa000]
> JavaCalls::call_helper(JavaValue*, methodHandle*,
> JavaCallArguments*, Thread*)+0x8f0 [0x000003530e4454a1]
> jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType,
> _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone
> .constprop.117]+0x1e1 (malloc=33783KB type=Internal #110876)
>
> Then we run it under heavy load for a few weeks and take another
> snapshot:
>
> [0x000003530e462f9a] JNIHandleBlock::allocate_block(Thread*)+0xaa
> [0x000003530e3f759a] JavaCallWrapper::JavaCallWrapper(methodHandle,
> Handle, JavaValue*, Thread*)+0x6a [0x000003530e3fa000]
> JavaCalls::call_helper(JavaValue*, methodHandle*,
> JavaCallArguments*, Thread*)+0x8f0 [0x000003530e4454a1]
> jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType,
> _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone
> .constprop.117]+0x1e1 (malloc=726749KB type=Internal #2385226)
>
> While other blocks also show some variation, none show growth like
> this one. When I do some math on the number (726749KB - 33783KB) /
> (2385226 – 110876) it comes down to a pretty even 312 bytes per
> allocation. And we leaked just under 700Mb. While not immediately
> problematic, this does not bode well for our customers who run this
> service for months.
>
> I’d like to avoid telling them they need to restart this service
> every two weeks to reclaim memory. Has anyone seen something like
> this? Any way it could be avoided?
That was some very good sleuthing on your part. 312 bytes per
allocation will indeed be very difficult to detect unless you are
really looking hard for it.
On 4/4/20 13:02, Mark Boon wrote:
> The connector of the webapp uses Http11NioProtocol. My
> understanding is it uses direct-byte-buffers backed by native
> memory for the Nio channels. I don't know for sure if that gets
> allocated through a JNI call, but that was my assumption.
This will definitely use Tomcat's NIO protocol which doesn't use the
APR connector. However, you still might be using tcnative to get the
crypto engine. Can you confirm the thread-naming convention of your
request-processing threads? They will tell you if JSSE or OpenSSL
(tcnative) is being used.
A few data points:
* No Tomcat code directly invokes jni_invoke_static(), but it might do
so indirectly through a variety of means.
* NIO does use buffers, but those buffers tend to be (a) fairly large
- -- on the order of kilobytes -- and (b) re-used for the life of the
request-processor thread.
It is very possible that there is a very small leak in Tomcat's
handling of NIO buffers. I think it's equally likely that there is a
bug in the JVM itself.
Are you able to try different JVM versions in your test? I would
recommend major-version changes, here. I thought I read somewhere that
Oracle re-wrote the implementation of the NIO API in a somewhat recent
Java release (Java 9?), but I can't seem to find that reference, now.
Are you able to try:
- - Java 8
- - Java 9/10/11/12
- - Java 13
- -chris
PS This bug report may be relevant:
https://bugs.openjdk.java.net/browse/JDK-8190395
The bug report says it's closed/incomplete, but they do mention a
312-byte leak with certain invocations.
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl6Lfx8ACgkQHPApP6U8
pFiGEQ/7BweMhSjALeMhEoAURv6NarT33Nb1ydHynOmDXnSXbczO+B6MC1j9QjHD
2sdYsPRLpi8holt2pl3dLxrsWE4gkK27e+2hwNN7568P/9S03m9VzpdciuwBCwfu
0acFtRs8iWcJO0jI29R438lt6w1CY6QGV2rasZKhkhaoBA8K9mSb0J02KUhDlpbn
oljBKA2k1oeqEDJpJoejsX7Zwgaf2aM96VZNds8Atly1WZhqj0nENwtU4yaIxel5
HyHrjOLsHMLPZeTDx/5pFs45qTGfFos1YF1lf99EDdKuX9qv+X+Dr2vr0RZW2iw8
5Oxh0EqAjMko9ysvjo2N7cnKK5fLNvAHRAGBIqDlAtcWF51vddeMASHjKYmM+/ha
pvN3/Dff65QXK74fgIk10Yqro1REudYcXTXT2+9WdycS06HOORndQaMs3l0qqaLG
057oHe9XuZs2th5imcS1D8zwaOmXo6jQmDYtGNEYk7pG8pfGfnHgnmVYCtGs067V
Rus6hMQAiUnEDgWVB/tYHZVCvzpR3nPVY5eckwComaSaSHD4p1nosRe7ILgaGJOu
fIxQGZ6PzgAAQZHgdqwkPeNLygZ3I46Nnliazdkejimyir6NLje8YW83LsWdlI5l
FlpdEkb92TA2TljDadkdHuUJGw+nFZmDla+AwZ4WPgI18PleBJs=
=PPJb
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
