Re: JNI memory leak?
Thanks Chris for taking the time.
As you point out, from the threads I can tell we're not using ARP as the names
al all starting with "jsse". AFAI could find out BouncyCastle is a pure Java
implementation, so that also can't be the cause.
Someone suggested PAMLibrary may be the culprit. So I started a thread that
makes continuous auth calls to the PAM library. Now there does seems to be an
indication memory is leaking very, very slowly. It seems to be roughly in line
with the number of auth failures. It looks like PAM throttles auth failures
though, hence it's taking such a long time for the evidence to mount.
So nothing to see here for this group. Just wanted to give a heads up.
Mark
On 4/6/20, 12:12 PM, "Christopher Schultz"
wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Mark,
On 4/3/20 21:48, Mark Boon wrote:
> For the past few months we’ve been trying to trace what looks like
> gradual memory creep. After some long-running experiments it seems
> due to memory leaking when jni_invoke_static(JNIEnv_*, JavaValue*,
> _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)
> is invoked. Somewhere.
>
> My environment is Tomcat running a proxy webapp. It does TLS
> termination, authentication and then forwards the call to local
> services. It doesn’t do much else, it’s a relatively small
> application.
>
> Some (possibly relevant) versions and config parameters: Tomcat
> 8.5 Java 8u241 (Oracle) Heap size = 360Mb MAX_ALLOC_ARENA=2
> MALLOC_TRIM_THRESHOLD_=250048 jdk.nio.maxCachedBufferSize=25600
>
> We couldn’t find any proof of memory leaking on the Java side. When
> we turn on NativeMemoryTracking=detail and we take a snapshot
> shortly after starting, we see (just one block shown):
>
> [0x03530e462f9a] JNIHandleBlock::allocate_block(Thread*)+0xaa
> [0x03530e3f759a] JavaCallWrapper::JavaCallWrapper(methodHandle,
> Handle, JavaValue*, Thread*)+0x6a [0x03530e3fa000]
> JavaCalls::call_helper(JavaValue*, methodHandle*,
> JavaCallArguments*, Thread*)+0x8f0 [0x03530e4454a1]
> jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType,
> _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone
> .constprop.117]+0x1e1 (malloc=33783KB type=Internal #110876)
>
> Then we run it under heavy load for a few weeks and take another
> snapshot:
>
> [0x03530e462f9a] JNIHandleBlock::allocate_block(Thread*)+0xaa
> [0x03530e3f759a] JavaCallWrapper::JavaCallWrapper(methodHandle,
> Handle, JavaValue*, Thread*)+0x6a [0x03530e3fa000]
> JavaCalls::call_helper(JavaValue*, methodHandle*,
> JavaCallArguments*, Thread*)+0x8f0 [0x03530e4454a1]
> jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType,
> _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone
> .constprop.117]+0x1e1 (malloc=726749KB type=Internal #2385226)
>
> While other blocks also show some variation, none show growth like
> this one. When I do some math on the number (726749KB - 33783KB) /
> (2385226 – 110876) it comes down to a pretty even 312 bytes per
> allocation. And we leaked just under 700Mb. While not immediately
> problematic, this does not bode well for our customers who run this
> service for months.
>
> I’d like to avoid telling them they need to restart this service
> every two weeks to reclaim memory. Has anyone seen something like
> this? Any way it could be avoided?
That was some very good sleuthing on your part. 312 bytes per
allocation will indeed be very difficult to detect unless you are
really looking hard for it.
On 4/4/20 13:02, Mark Boon wrote:
> The connector of the webapp uses Http11NioProtocol. My
> understanding is it uses direct-byte-buffers backed by native
> memory for the Nio channels. I don't know for sure if that gets
> allocated through a JNI call, but that was my assumption.
This will definitely use Tomcat's NIO protocol which doesn't use the
APR connector. However, you still might be using tcnative to get the
crypto engine. Can you confirm the thread-naming convention of your
request-processing threads? They will tell you if JSSE or OpenSSL
(tcnative) is being used.
A few data points:
* No Tomcat code directly invokes jni_invoke_static(), but it might do
so indirectly through a variety of means.
* NIO does use buffers, but those buffers tend to be (a) fairly large
- -- on the order of kilobytes -- and (b) re-used for the life of the
request-processor thread.
It is very possible that there is a very small leak in Tomcat's
handling of NIO buf
Re: JNI memory leak?
I don't have 'proof' Tomcat is to blame. Hence the question-mark. All I have managed is narrow it down to this NMT data, which is not very informative. I hoped anyone could give me an idea how or where to investigate further. Or if someone had run into this before. The connector of the webapp uses Http11NioProtocol. My understanding is it uses direct-byte-buffers backed by native memory for the Nio channels. I don't know for sure if that gets allocated through a JNI call, but that was my assumption. I did not consider trying Mission Control or jvisualvm. Isn't Mission Control for embedded Java? And AFAIK, jvisualvm is for profiling Java memory usage and underneath uses tools like jmap, jstat and jcmd. Through GC logs and jmap heap-dumps I can confidently say there's no memory leak on the Java side. The NMT data shown comes from jcmd. No type grows beyond control and full GC always returns to the same baseline for the heap. Anyway, the Java heap is only 360Mb and this memory-block created by jni_invoke_static has grown to 700Mb by itself. And I see no out-of-memory messages. The only hint of this happening is that the RES memory of the Tomcat process keeps growing over time, as shown by 'top'. And it seems GC is getting slower over time, but the customers haven't noticed it yet. (This is after we switched to ParallelGC. We did see considerable slow-down when using G1GC in the ref-processing, but we couldn't figure out why. It would slow to a crawl before the memory leak became obvious.) Anyway, I was mostly fishing for hints or tips that could help me figure this out or avoid it. The application is simple to the point I'm hard-pressed to think of any other part making JNI calls. The only library I can think of using JNI is BouncyCastle doing the SSL encryption/decryption, so maybe I'll switch my focus there. Thanks for taking the time to think along. Mark On 4/4/20, 5:50 AM, "calder" wrote: On Fri, Apr 3, 2020 at 8:48 PM Mark Boon wrote: > > For the past few months we’ve been trying to trace what looks like gradual memory creep. After some long-running experiments it seems due to memory leaking when > jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) is invoked. Somewhere. > > My environment is Tomcat running a proxy webapp. It does TLS termination, authentication and then forwards the call to local services. It doesn’t do much else, it’s a relatively small application. > > Some (possibly relevant) versions and config parameters: > Tomcat 8.5 > Java 8u241 (Oracle) > Heap size = 360Mb > MAX_ALLOC_ARENA=2 > MALLOC_TRIM_THRESHOLD_=250048 > jdk.nio.maxCachedBufferSize=25600 > > We couldn’t find any proof of memory leaking on the Java side. > When we turn on NativeMemoryTracking=detail and we take a snapshot shortly after starting, we see (just one block shown): > > [0x03530e462f9a] JNIHandleBlock::allocate_block(Thread*)+0xaa > [0x03530e3f759a] JavaCallWrapper::JavaCallWrapper(methodHandle, Handle, JavaValue*, Thread*)+0x6a > [0x03530e3fa000] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x8f0 > [0x03530e4454a1] jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone .constprop.117]+0x1e1 > (malloc=33783KB type=Internal #110876) > > Then we run it under heavy load for a few weeks and take another snapshot: > > [0x03530e462f9a] JNIHandleBlock::allocate_block(Thread*)+0xaa > [0x03530e3f759a] JavaCallWrapper::JavaCallWrapper(methodHandle, Handle, JavaValue*, Thread*)+0x6a > [0x03530e3fa000] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x8f0 > [0x03530e4454a1] jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone .constprop.117]+0x1e1 > (malloc=726749KB type=Internal #2385226) > > While other blocks also show some variation, none show growth like this one. When I do some math on the number (726749KB - 33783KB) / (2385226 – 110876) it comes down to a pretty even 312 bytes per allocation. > And we leaked just under 700Mb. While not immediately problematic, this does not bode well for our customers who run this service for months. > > I’d like to avoid telling them they need to restart this service every two weeks to reclaim memory. Has anyone seen something like this? Any way it could be avoided? I'm a bit confused. Your stated title is "JNI Memory Leak?" Tomcat, to my intimate knowledge, does not use JNI (correct m
JNI memory leak?
For the past few months we’ve been trying to trace what looks like gradual memory creep. After some long-running experiments it seems due to memory leaking when jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) is invoked. Somewhere. My environment is Tomcat running a proxy webapp. It does TLS termination, authentication and then forwards the call to local services. It doesn’t do much else, it’s a relatively small application. Some (possibly relevant) versions and config parameters: Tomcat 8.5 Java 8u241 (Oracle) Heap size = 360Mb MAX_ALLOC_ARENA=2 MALLOC_TRIM_THRESHOLD_=250048 jdk.nio.maxCachedBufferSize=25600 We couldn’t find any proof of memory leaking on the Java side. When we turn on NativeMemoryTracking=detail and we take a snapshot shortly after starting, we see (just one block shown): [0x03530e462f9a] JNIHandleBlock::allocate_block(Thread*)+0xaa [0x03530e3f759a] JavaCallWrapper::JavaCallWrapper(methodHandle, Handle, JavaValue*, Thread*)+0x6a [0x03530e3fa000] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x8f0 [0x03530e4454a1] jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone .constprop.117]+0x1e1 (malloc=33783KB type=Internal #110876) Then we run it under heavy load for a few weeks and take another snapshot: [0x03530e462f9a] JNIHandleBlock::allocate_block(Thread*)+0xaa [0x03530e3f759a] JavaCallWrapper::JavaCallWrapper(methodHandle, Handle, JavaValue*, Thread*)+0x6a [0x03530e3fa000] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x8f0 [0x03530e4454a1] jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone .constprop.117]+0x1e1 (malloc=726749KB type=Internal #2385226) While other blocks also show some variation, none show growth like this one. When I do some math on the number (726749KB - 33783KB) / (2385226 – 110876) it comes down to a pretty even 312 bytes per allocation. And we leaked just under 700Mb. While not immediately problematic, this does not bode well for our customers who run this service for months. I’d like to avoid telling them they need to restart this service every two weeks to reclaim memory. Has anyone seen something like this? Any way it could be avoided? Mark Boon
Re: Can Tomcat log handshake failures, and where?
Hi Mark, Well, anything is 100% better than nothing. Is this "127.0.0.1 - - [31/Jul/2019:16:45:16 +0100] "-" 400 -" going to be followed by any reason or error-code that can point to the reason of failure? Anything that distinguishes it from a 'regular' 400 error originating after the handshake? I'd have to pass it by the compliance experts, but maybe even just this would be enough to convince them I don't need to use the javax.net.debug=ssl:handshake sledge-hammer. What version will this be in? Mark Boon From: Mark Thomas Sent: Wednesday, July 31, 2019 8:47 AM To: users@tomcat.apache.org Subject: Re: Can Tomcat log handshake failures, and where? On 30/07/2019 08:28, Mark Thomas wrote: > Generally, processing needs to get as far as presenting a request line > before something is added to the access logs. We could look at expanding > the access logging to include connections that are dropped earlier but > that might be a sufficiently invasive change that it needs to wait until > Tomcat 10. I've done some work on this and it looks promising. The end result is entries like this in the access log for a failed TLS handshake: 127.0.0.1 - - [31/Jul/2019:16:45:16 +0100] "-" 400 - 127.0.0.1 - - [31/Jul/2019:16:45:16 +0100] "-" 400 - 127.0.0.1 - - [31/Jul/2019:16:45:17 +0100] "-" 400 - 127.0.0.1 - - [31/Jul/2019:16:45:17 +0100] "-" 400 - Does this meet your requirement? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Can Tomcat log handshake failures, and where?
Christopher, No, I don't need to log THAT failure. But I do need to log handshake failures where the failure to connect was a server-side decision. So (apparently) I do need to log cases like a handshake failure in case both sides couldn't agree on a cipher, as detailed in my question. As much as I'd like to declare handshake failures "not my problem", it doesn't help us sell to governments that require this case to be logged. And it's not as if it's technically impossible, as clearly it can be reported by setting javax.net.debug=ssl:handshake. But I want to log just the failure, and not the fire-hose amount of information this gives me for every successful handshake. Mark From: Christopher Schultz Sent: Tuesday, July 30, 2019 8:13 AM To: users@tomcat.apache.org Subject: Re: Can Tomcat log handshake failures, and where? -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 7/29/19 17:45, Mark Boon wrote: > Apparently for compliance reasons we're required to log any failed > connection attempt. So I'd like to know if and how I can get > Tomcat to emit such information. I'd try to get some clarification on that requirement. For example, if a client tries to connect and they have a network error on their end (e.g. ISP fails), are you somehow required to log THAT failure? TLS handshake failures should fall under the category of "not my problem" and you really shouldn't have to log them. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl1AXq4ACgkQHPApP6U8 pFjMIA//ctjGwEnC+yRH5awk/uPx8eaiUy63/MVOVf45NAPOs9DoCsHd3X27Mzek lKhwxfc4q8OZICfWyv+nk/VD/aAUwc1LV4ju2tq/1Qe87QQ/KnOs1ySsgTywkOE8 YEoDzVRhKw5mMBjS3Avkw+xTi5eWpUpi/fhStUuntRme+9F0JtXyuRj4V8lkpssa MOg0krWOd4jn0ngxZ1MkCx8Ybh0keKcBpyDBo3jqeZ8PUA4jTYAeVXN5eah7Hi2A kpNWi4ny7f988ZDbufXiYJxP7J3DiZ71+peKxr1NS1CDXrCBMg0xO4j9rcxjJpCY pIaJrttzJzBMjQrmUAPPfrgIYo4LCiwX6K5YFDzifFucBMCtYtHiBZBEjjLh5JJR HKU0jU0pMjru6HaKab76O85nTjMrl5P9ouvbxTPUNxtlEreFH+4cNUyf4CKlOjt2 zkK9RXeJwzpmOSlK2BmW6sC9UpCeFrpIvzBTCdhZ2EGA/ORaMK0Evz+VMSWbTvhs GMi1DIDIZh1X0Vzed9gncNKibjKMwwdEnly5MybI4qXfoPGv8Z+l6T6pjQd6hA+P c2iKjSiPdUHVSoL8iwYzVa7Yqrs+2rqnqVRA/RzX5IU5Jqj6C7A7dPxMWAirAowX M3/4CoqcK+LXqnTY/yZNQqquWrf155KHNTDuRry2CexaBNmL80A= =+jM7 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: How to configure Tomcat for OCSP stapling?
Hi Mark Thomas, Thanks for the pointer. However, that comment you posted there is 4 years old. Any news on the matter since then at all? Has OCSP Stapling support for NIO connectors since made it into Java 9? Mark Boon On 5/30/18, 12:46 AM, "Mark Thomas" wrote: On 29/05/18 00:22, Mark Boon wrote: > My company asked to enable OCSP stapling for our Tomcat server. I found > the documentation about configuring a Tomcat OCSP Connector here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__tomcat.apache.org_tomcat-2D8.5-2Ddoc_ssl-2Dhowto.html-23Configuring-5FOCSP-5FConnector=DwIDaQ=uilaK90D4TOVoH58JNXRgQ=_kwXikaSZUUarF811P_o9Q=c9Hngb286HQ3waldNl7R5ScNf7kDJHlpVrzQqNdzrqA=Su_lL2hasSRZW0qqEbINeT1Cg6YL5lUjcf18mJXo0kA= > > > > However, if I’m not mistaken those are instructions for how to set up an > OCSP responder. But I think in my case, the OCSP responder is the CA > that issued the certificate. What I need is to instruct Tomcat so that > it makes the call to the OCSP responder that is specified in the CA > signed certificate and ‘staples’ the resulting ticket to the certificate > before presenting it to the client. > > > > Does anyone know of a place with instructions how to do something like > this? Or possibly I’m not quite understanding the process of OCSP > stapling, in which case any pointers on what it means and how it works > with Tomcat would be much appreciated. https://urldefense.proofpoint.com/v2/url?u=https-3A__bz.apache.org_bugzilla_show-5Fbug.cgi-3Fid-3D56148=DwIDaQ=uilaK90D4TOVoH58JNXRgQ=_kwXikaSZUUarF811P_o9Q=c9Hngb286HQ3waldNl7R5ScNf7kDJHlpVrzQqNdzrqA=O87uG9CLTZkFihZww0qxd5jkZV6AaWhZ_KE5Kk2JKhU= Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
How to configure Tomcat for OCSP stapling?
My company asked to enable OCSP stapling for our Tomcat server. I found the documentation about configuring a Tomcat OCSP Connector here: https://tomcat.apache.org/tomcat-8.5-doc/ssl-howto.html#Configuring_OCSP_Connector However, if I’m not mistaken those are instructions for how to set up an OCSP responder. But I think in my case, the OCSP responder is the CA that issued the certificate. What I need is to instruct Tomcat so that it makes the call to the OCSP responder that is specified in the CA signed certificate and ‘staples’ the resulting ticket to the certificate before presenting it to the client. Does anyone know of a place with instructions how to do something like this? Or possibly I’m not quite understanding the process of OCSP stapling, in which case any pointers on what it means and how it works with Tomcat would be much appreciated. Mark Boon Staff Engineer mb...@vmware.com 3401 Hillview Avenue, Palo Alto, CA 94304 650.123.4567 Office 808.234.4892 Mobile [e]<http://www.vmware.com/>
Re: Insert key-store implementation into Tomcat Connector
Thank you for the information. Mark Boon On 5/2/18, 3:35 AM, "Mark Thomas" <ma...@apache.org> wrote: On 02/05/18 01:48, Mark Boon wrote: > In the Tomcat TLS Connecter configuration, there’s the trustManagerClassName that can be set to a Java implementation of the X509TrustManager interface. There’s also a configuration called keystoreFile from which it will read the certificate-key pair to set up the SSL connection. I was wondering if there’s also a way to configure a class that will provide the SSL certificate? My company would like to plug in their own mechanism to store and retrieve certificates, rather than the Java Key Store. > > I have seen references to a keystoreProvider, but I have been unable to find anything that provides an example how that is to be used, so I’m not sure that serves for what I’m looking for. > > Any pointer to how that could be accomplished would be highly appreciated. Writing a custom Keystore provider would be the most portable solution. https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.oracle.com_javase_8_docs_technotes_guides_security_crypto_HowToImplAProvider.html=DwIDaQ=uilaK90D4TOVoH58JNXRgQ=_kwXikaSZUUarF811P_o9Q=8HmxxPHbEVv5mWXdTWFydzyBxICcZhOKruAGLF3F5xU=Y4hc81hr--crgkyPquO5rADDFJ-2uJTEv3oC6WtbCKU= has some pointers. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Insert key-store implementation into Tomcat Connector
In the Tomcat TLS Connecter configuration, there’s the trustManagerClassName that can be set to a Java implementation of the X509TrustManager interface. There’s also a configuration called keystoreFile from which it will read the certificate-key pair to set up the SSL connection. I was wondering if there’s also a way to configure a class that will provide the SSL certificate? My company would like to plug in their own mechanism to store and retrieve certificates, rather than the Java Key Store. I have seen references to a keystoreProvider, but I have been unable to find anything that provides an example how that is to be used, so I’m not sure that serves for what I’m looking for. Any pointer to how that could be accomplished would be highly appreciated. Mark Boon
