On Wed, 2016-03-23 at 18:54 +0000, Harold Rosenberg wrote:
> I am seeing unexpected behavior in the memory usage of a program that uses
> the Async Http Client. The issue arises when I switch from using exclusively
> HTTP connections to exclusively HTTPS connections. In the latter case, there
> is a large amount of memory used for byte arrays that seem to be related to
> SSL processing, and that aren’t ever cleaned up by garbage collection.
>
> My question is whether this is expected due to differences between SSL and
> non-SSL processing, or whether I am failing to properly clean up my SSL
> connections.
>
> The program is a workload driver for a performance benchmark. It simulates
> user activity to a web service. Each simulated user has about four active
> connections to the service. Normally the connections are all over SSL
> (really TLS), but for testing purposes there is a switch to use only non-SSL
> connections. The configuration of the HTTP client is identical for in both
> cases.
>
> I gathered garbage-collection data with and without TLS for runs with 7,500
> simulated users, which would average about 30,000 active connections. A user
> would be active for about five minutes, at which point the they log out and a
> new user logs in. While the user is logged in the connections will
> occasionally be idle and thus closed due to keepalive timeout. The runs were
> two hours long and the data was collected during a steady-state. Data is
> generated in the heap at about the same rate in both cases, but when running
> with HTTPS there is about 3.2GB of live data left in the heap after a full
> garbage-collection, while with just HTTP there is only about 890MB. The
> amount of live data in the heap doesn’t grow once it reaches the 3.2GB mark,
> so it isn’t a memory leak, but it is definitely proportional to the number of
> active connections. When I do the same run with 15,000 users, at
> steady-state the live data in the heap after a full GC is about 6.4GB.
>
> Of the 3.2GB of live data, about 2.8GB is in 876,041 instances of byte[].
> Almost all of the instances seem to be related to SSL processing. Here is
> the path to the nearest GC root for a few random instances:
>
Hi Harold,
Could you try using ReleasableSSLBufferManagementStrategy instead of
default PermanentSSLBufferManagementStrategy?
---
DefaultHostnameVerifier hostnameVerifier = new
DefaultHostnameVerifier(PublicSuffixMatcherLoader.getDefault());
Registry<SchemeIOSessionStrategy> registry =
RegistryBuilder.<SchemeIOSessionStrategy>create()
.register("http", NoopIOSessionStrategy.INSTANCE)
.register("https", new SchemeIOSessionStrategy() {
@Override
public boolean isLayeringRequired() {
return true;
}
@Override
public IOSession upgrade(final HttpHost host, final IOSession
iosession) throws IOException {
final SSLIOSession ssliosession = new SSLIOSession(
iosession,
SSLMode.CLIENT,
host,
SSLContexts.createSystemDefault(),
new SSLSetupHandler() {
@Override
public void initalize(final SSLEngine sslengine)
throws SSLException {
}
@Override
public void verify(
final IOSession iosession,
final SSLSession sslsession) throws
SSLException {
if
(!hostnameVerifier.verify(host.getHostName(), sslsession)) {
final Certificate[] certs =
sslsession.getPeerCertificates();
final X509Certificate x509 =
(X509Certificate) certs[0];
final X500Principal x500Principal =
x509.getSubjectX500Principal();
throw new SSLPeerUnverifiedException("Host
name '" + host.getHostName() + "' does not match " +
"the certificate subject provided
by the peer (" + x500Principal.toString() + ")");
}
}
},
new ReleasableSSLBufferManagementStrategy());
iosession.setAttribute(SSLIOSession.SESSION_KEY, ssliosession);
ssliosession.initialize();
return ssliosession;
}
})
.build();
ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor();
PoolingNHttpClientConnectionManager cm = new
PoolingNHttpClientConnectionManager(ioReactor, registry);
CloseableHttpAsyncClient httpclient = HttpAsyncClients.createMinimal(cm);
try {
httpclient.start();
HttpGet request = new HttpGet("https://verisign.com/";);
Future<HttpResponse> future = httpclient.execute(request, null);
HttpResponse response = future.get();
System.out.println("Response: " + response.getStatusLine());
System.out.println("Shutting down");
} finally {
httpclient.close();
}
System.out.println("Done");
---
Oleg
> this - value: byte[] #483799
> <- hb - class: java.nio.HeapByteBuffer, value: byte[] #483799
> <- buffer - class:
> org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer,
> value: java.nio.HeapByteBuffer #89456
> <- outPlain - class: org.apache.http.nio.reactor.ssl.SSLIOSession,
> value:
> org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer
> #59632
> <- bufferStatus - class:
> org.apache.http.impl.nio.reactor.IOSessionImpl, value:
> org.apache.http.nio.reactor.ssl.SSLIOSession #14910
> <- attachment - class: sun.nio.ch.SelectionKeyImpl, value:
> org.apache.http.impl.nio.reactor.IOSessionImpl #14910
> <- value - class: java.util.HashMap$Entry, value:
> sun.nio.ch.SelectionKeyImpl #14910
> <- [355] - class: java.util.HashMap$Entry[], value:
> java.util.HashMap$Entry #197065
> <- table - class: java.util.HashMap, value:
> java.util.HashMap$Entry[] #4612
> <- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl,
> value: java.util.HashMap #10016
>
> this - value: byte[] #60139
> <- key - class:
> com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey, value:
> byte[] #60139
> <- masterSecret - class: sun.security.ssl.SSLSessionImpl, value:
> com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey #4827
> <- sess - class: sun.security.ssl.SSLEngineImpl, value:
> sun.security.ssl.SSLSessionImpl #4829
> <- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession,
> value: sun.security.ssl.SSLEngineImpl #20572
> <- bufferStatus - class:
> org.apache.http.impl.nio.reactor.IOSessionImpl, value:
> org.apache.http.nio.reactor.ssl.SSLIOSession #20572
> <- attachment - class: sun.nio.ch.SelectionKeyImpl, value:
> org.apache.http.impl.nio.reactor.IOSessionImpl #20572
> <- value - class: java.util.HashMap$Entry, value:
> sun.nio.ch.SelectionKeyImpl #20572
> <- [1182] - class: java.util.HashMap$Entry[], value:
> java.util.HashMap$Entry #254234
> <- table - class: java.util.HashMap, value:
> java.util.HashMap$Entry[] #6977
> <- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl,
> value: java.util.HashMap #9595
>
> this - value: byte[] #71913
> <- iv - class: com.sun.crypto.provider.CipherBlockChaining, value:
> byte[] #71913
> <- cipher - class: com.sun.crypto.provider.CipherCore, value:
> com.sun.crypto.provider.CipherBlockChaining #3709
> <- core - class: com.sun.crypto.provider.AESCipher$General, value:
> com.sun.crypto.provider.CipherCore #3709
> <- spi - class: javax.crypto.Cipher, value:
> com.sun.crypto.provider.AESCipher$General #3709
> <- cipher - class: sun.security.ssl.CipherBox, value:
> javax.crypto.Cipher #3709
> <- readCipher - class: sun.security.ssl.SSLEngineImpl, value:
> sun.security.ssl.CipherBox #3710
> <- sslEngine - class:
> org.apache.http.nio.reactor.ssl.SSLIOSession, value:
> sun.security.ssl.SSLEngineImpl #1855
> <- bufferStatus - class:
> org.apache.http.impl.nio.reactor.IOSessionImpl, value:
> org.apache.http.nio.reactor.ssl.SSLIOSession #1855
> <- key - class: java.util.HashMap$Entry, value:
> org.apache.http.impl.nio.reactor.IOSessionImpl #1855
> <- [4000] - class: java.util.HashMap$Entry[], value:
> java.util.HashMap$Entry #71715
> <- table - class: java.util.HashMap, value:
> java.util.HashMap$Entry[] #24138
> <- map - class: java.util.HashSet, value: java.util.HashMap
> #8449
> <- c - class: java.util.Collections$SynchronizedSet, value:
> java.util.HashSet #3312
> <- sessions (Java frame) - class:
> org.apache.http.impl.nio.reactor.BaseIOReactor, value:
> java.util.Collections$SynchronizedSet #6
>
> I can give more details about the configuration of the HTTP Client and the
> SSL sessions if it would be helpful, but I don’t want to give too much info
> if this is expected behavior.
>
> Any advice would be appreciated.
>
> Thanks,
>
> Hal
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
