Using the ReleasableSSLBufferManagementStrategy gave a nice improvement. For
the same run with 7500 simulated users, the change reduced the average live
data in the heap after a full GC from 3.2GB to about 1.5GB. The amount of live
data is still proportional to the number of connections (about 3GB of live data
for a run with 15,000 simulated users), but that is to be expected.
Thanks for the help.
Hal
On 3/23/16, 3:59 PM, "Oleg Kalnichevski" <ol...@apache.org> wrote:
>On Wed, 2016-03-23 at 18:54 +0000, Harold Rosenberg wrote:
>> I am seeing unexpected behavior in the memory usage of a program that uses
>> the Async Http Client. The issue arises when I switch from using
>> exclusively HTTP connections to exclusively HTTPS connections. In the
>> latter case, there is a large amount of memory used for byte arrays that
>> seem to be related to SSL processing, and that aren’t ever cleaned up by
>> garbage collection.
>>
>> My question is whether this is expected due to differences between SSL and
>> non-SSL processing, or whether I am failing to properly clean up my SSL
>> connections.
>>
>> The program is a workload driver for a performance benchmark. It simulates
>> user activity to a web service. Each simulated user has about four active
>> connections to the service. Normally the connections are all over SSL
>> (really TLS), but for testing purposes there is a switch to use only non-SSL
>> connections. The configuration of the HTTP client is identical for in both
>> cases.
>>
>> I gathered garbage-collection data with and without TLS for runs with 7,500
>> simulated users, which would average about 30,000 active connections. A
>> user would be active for about five minutes, at which point the they log out
>> and a new user logs in. While the user is logged in the connections will
>> occasionally be idle and thus closed due to keepalive timeout. The runs
>> were two hours long and the data was collected during a steady-state. Data
>> is generated in the heap at about the same rate in both cases, but when
>> running with HTTPS there is about 3.2GB of live data left in the heap after
>> a full garbage-collection, while with just HTTP there is only about 890MB.
>> The amount of live data in the heap doesn’t grow once it reaches the 3.2GB
>> mark, so it isn’t a memory leak, but it is definitely proportional to the
>> number of active connections. When I do the same run with 15,000 users, at
>> steady-state the live data in the heap after a full GC is about 6.4GB.
>>
>> Of the 3.2GB of live data, about 2.8GB is in 876,041 instances of byte[].
>> Almost all of the instances seem to be related to SSL processing. Here is
>> the path to the nearest GC root for a few random instances:
>>
>
>Hi Harold,
>
>Could you try using ReleasableSSLBufferManagementStrategy instead of
>default PermanentSSLBufferManagementStrategy?
>
>---
>DefaultHostnameVerifier hostnameVerifier = new
>DefaultHostnameVerifier(PublicSuffixMatcherLoader.getDefault());
>Registry<SchemeIOSessionStrategy> registry =
>RegistryBuilder.<SchemeIOSessionStrategy>create()
> .register("http", NoopIOSessionStrategy.INSTANCE)
> .register("https", new SchemeIOSessionStrategy() {
> @Override
> public boolean isLayeringRequired() {
> return true;
> }
>
> @Override
> public IOSession upgrade(final HttpHost host, final IOSession
> iosession) throws IOException {
> final SSLIOSession ssliosession = new SSLIOSession(
> iosession,
> SSLMode.CLIENT,
> host,
> SSLContexts.createSystemDefault(),
> new SSLSetupHandler() {
>
> @Override
> public void initalize(final SSLEngine sslengine)
> throws SSLException {
> }
>
> @Override
> public void verify(
> final IOSession iosession,
> final SSLSession sslsession) throws
> SSLException {
> if
> (!hostnameVerifier.verify(host.getHostName(), sslsession)) {
> final Certificate[] certs =
> sslsession.getPeerCertificates();
> final X509Certificate x509 =
> (X509Certificate) certs[0];
> final X500Principal x500Principal =
> x509.getSubjectX500Principal();
> throw new SSLPeerUnverifiedException("Host
> name '" + host.getHostName() + "' does not match " +
> "the certificate subject provided
> by the peer (" + x500Principal.toString() + ")");
> }
> }
>
> },
> new ReleasableSSLBufferManagementStrategy());
> iosession.setAttribute(SSLIOSession.SESSION_KEY, ssliosession);
> ssliosession.initialize();
> return ssliosession;
> }
> })
> .build();
>
>ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor();
>PoolingNHttpClientConnectionManager cm = new
>PoolingNHttpClientConnectionManager(ioReactor, registry);
>CloseableHttpAsyncClient httpclient = HttpAsyncClients.createMinimal(cm);
>try {
> httpclient.start();
> HttpGet request = new
> HttpGet("https://urldefense.proofpoint.com/v2/url?u=https-3A__verisign.com_&d=BQIDaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=9uYCHvq1QDKcMAIXCt6iT2etdTpbRn0toGoRKE6QxRo&m=HkvNV6bDDfPghsn_MF5SgU7KlMhzmAGjcfgAmviJCSs&s=w1-QtI1DaVx22Oo8UumpY0uvEblsK5t55MihZ_UwmPM&e=
> ");
> Future<HttpResponse> future = httpclient.execute(request, null);
> HttpResponse response = future.get();
> System.out.println("Response: " + response.getStatusLine());
> System.out.println("Shutting down");
>} finally {
> httpclient.close();
>}
>System.out.println("Done");
>---
>
>Oleg
>
>> this - value: byte[] #483799
>> <- hb - class: java.nio.HeapByteBuffer, value: byte[] #483799
>> <- buffer - class:
>> org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer,
>> value: java.nio.HeapByteBuffer #89456
>> <- outPlain - class: org.apache.http.nio.reactor.ssl.SSLIOSession,
>> value:
>> org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer
>> #59632
>> <- bufferStatus - class:
>> org.apache.http.impl.nio.reactor.IOSessionImpl, value:
>> org.apache.http.nio.reactor.ssl.SSLIOSession #14910
>> <- attachment - class: sun.nio.ch.SelectionKeyImpl, value:
>> org.apache.http.impl.nio.reactor.IOSessionImpl #14910
>> <- value - class: java.util.HashMap$Entry, value:
>> sun.nio.ch.SelectionKeyImpl #14910
>> <- [355] - class: java.util.HashMap$Entry[], value:
>> java.util.HashMap$Entry #197065
>> <- table - class: java.util.HashMap, value:
>> java.util.HashMap$Entry[] #4612
>> <- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl,
>> value: java.util.HashMap #10016
>>
>> this - value: byte[] #60139
>> <- key - class:
>> com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey, value:
>> byte[] #60139
>> <- masterSecret - class: sun.security.ssl.SSLSessionImpl, value:
>> com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey #4827
>> <- sess - class: sun.security.ssl.SSLEngineImpl, value:
>> sun.security.ssl.SSLSessionImpl #4829
>> <- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession,
>> value: sun.security.ssl.SSLEngineImpl #20572
>> <- bufferStatus - class:
>> org.apache.http.impl.nio.reactor.IOSessionImpl, value:
>> org.apache.http.nio.reactor.ssl.SSLIOSession #20572
>> <- attachment - class: sun.nio.ch.SelectionKeyImpl, value:
>> org.apache.http.impl.nio.reactor.IOSessionImpl #20572
>> <- value - class: java.util.HashMap$Entry, value:
>> sun.nio.ch.SelectionKeyImpl #20572
>> <- [1182] - class: java.util.HashMap$Entry[], value:
>> java.util.HashMap$Entry #254234
>> <- table - class: java.util.HashMap, value:
>> java.util.HashMap$Entry[] #6977
>> <- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl,
>> value: java.util.HashMap #9595
>>
>> this - value: byte[] #71913
>> <- iv - class: com.sun.crypto.provider.CipherBlockChaining, value:
>> byte[] #71913
>> <- cipher - class: com.sun.crypto.provider.CipherCore, value:
>> com.sun.crypto.provider.CipherBlockChaining #3709
>> <- core - class: com.sun.crypto.provider.AESCipher$General, value:
>> com.sun.crypto.provider.CipherCore #3709
>> <- spi - class: javax.crypto.Cipher, value:
>> com.sun.crypto.provider.AESCipher$General #3709
>> <- cipher - class: sun.security.ssl.CipherBox, value:
>> javax.crypto.Cipher #3709
>> <- readCipher - class: sun.security.ssl.SSLEngineImpl, value:
>> sun.security.ssl.CipherBox #3710
>> <- sslEngine - class:
>> org.apache.http.nio.reactor.ssl.SSLIOSession, value:
>> sun.security.ssl.SSLEngineImpl #1855
>> <- bufferStatus - class:
>> org.apache.http.impl.nio.reactor.IOSessionImpl, value:
>> org.apache.http.nio.reactor.ssl.SSLIOSession #1855
>> <- key - class: java.util.HashMap$Entry, value:
>> org.apache.http.impl.nio.reactor.IOSessionImpl #1855
>> <- [4000] - class: java.util.HashMap$Entry[], value:
>> java.util.HashMap$Entry #71715
>> <- table - class: java.util.HashMap, value:
>> java.util.HashMap$Entry[] #24138
>> <- map - class: java.util.HashSet, value: java.util.HashMap
>> #8449
>> <- c - class: java.util.Collections$SynchronizedSet, value:
>> java.util.HashSet #3312
>> <- sessions (Java frame) - class:
>> org.apache.http.impl.nio.reactor.BaseIOReactor, value:
>> java.util.Collections$SynchronizedSet #6
>>
>> I can give more details about the configuration of the HTTP Client and the
>> SSL sessions if it would be helpful, but I don’t want to give too much info
>> if this is expected behavior.
>>
>> Any advice would be appreciated.
>>
>> Thanks,
>>
>> Hal
>>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
>For additional commands, e-mail: httpclient-users-h...@hc.apache.org
>
