On Wed, 2016-03-23 at 18:54 +0000, Harold Rosenberg wrote: > I am seeing unexpected behavior in the memory usage of a program that uses > the Async Http Client. The issue arises when I switch from using exclusively > HTTP connections to exclusively HTTPS connections. In the latter case, there > is a large amount of memory used for byte arrays that seem to be related to > SSL processing, and that aren’t ever cleaned up by garbage collection. > > My question is whether this is expected due to differences between SSL and > non-SSL processing, or whether I am failing to properly clean up my SSL > connections. > > The program is a workload driver for a performance benchmark. It simulates > user activity to a web service. Each simulated user has about four active > connections to the service. Normally the connections are all over SSL > (really TLS), but for testing purposes there is a switch to use only non-SSL > connections. The configuration of the HTTP client is identical for in both > cases. > > I gathered garbage-collection data with and without TLS for runs with 7,500 > simulated users, which would average about 30,000 active connections. A user > would be active for about five minutes, at which point the they log out and a > new user logs in. While the user is logged in the connections will > occasionally be idle and thus closed due to keepalive timeout. The runs were > two hours long and the data was collected during a steady-state. Data is > generated in the heap at about the same rate in both cases, but when running > with HTTPS there is about 3.2GB of live data left in the heap after a full > garbage-collection, while with just HTTP there is only about 890MB. The > amount of live data in the heap doesn’t grow once it reaches the 3.2GB mark, > so it isn’t a memory leak, but it is definitely proportional to the number of > active connections. When I do the same run with 15,000 users, at > steady-state the live data in the heap after a full GC is about 6.4GB. > > Of the 3.2GB of live data, about 2.8GB is in 876,041 instances of byte[]. > Almost all of the instances seem to be related to SSL processing. Here is > the path to the nearest GC root for a few random instances: >
Hi Harold, Could you try using ReleasableSSLBufferManagementStrategy instead of default PermanentSSLBufferManagementStrategy? --- DefaultHostnameVerifier hostnameVerifier = new DefaultHostnameVerifier(PublicSuffixMatcherLoader.getDefault()); Registry<SchemeIOSessionStrategy> registry = RegistryBuilder.<SchemeIOSessionStrategy>create() .register("http", NoopIOSessionStrategy.INSTANCE) .register("https", new SchemeIOSessionStrategy() { @Override public boolean isLayeringRequired() { return true; } @Override public IOSession upgrade(final HttpHost host, final IOSession iosession) throws IOException { final SSLIOSession ssliosession = new SSLIOSession( iosession, SSLMode.CLIENT, host, SSLContexts.createSystemDefault(), new SSLSetupHandler() { @Override public void initalize(final SSLEngine sslengine) throws SSLException { } @Override public void verify( final IOSession iosession, final SSLSession sslsession) throws SSLException { if (!hostnameVerifier.verify(host.getHostName(), sslsession)) { final Certificate[] certs = sslsession.getPeerCertificates(); final X509Certificate x509 = (X509Certificate) certs[0]; final X500Principal x500Principal = x509.getSubjectX500Principal(); throw new SSLPeerUnverifiedException("Host name '" + host.getHostName() + "' does not match " + "the certificate subject provided by the peer (" + x500Principal.toString() + ")"); } } }, new ReleasableSSLBufferManagementStrategy()); iosession.setAttribute(SSLIOSession.SESSION_KEY, ssliosession); ssliosession.initialize(); return ssliosession; } }) .build(); ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor(); PoolingNHttpClientConnectionManager cm = new PoolingNHttpClientConnectionManager(ioReactor, registry); CloseableHttpAsyncClient httpclient = HttpAsyncClients.createMinimal(cm); try { httpclient.start(); HttpGet request = new HttpGet("https://verisign.com/"); Future<HttpResponse> future = httpclient.execute(request, null); HttpResponse response = future.get(); System.out.println("Response: " + response.getStatusLine()); System.out.println("Shutting down"); } finally { httpclient.close(); } System.out.println("Done"); --- Oleg > this - value: byte[] #483799 > <- hb - class: java.nio.HeapByteBuffer, value: byte[] #483799 > <- buffer - class: > org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer, > value: java.nio.HeapByteBuffer #89456 > <- outPlain - class: org.apache.http.nio.reactor.ssl.SSLIOSession, > value: > org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer > #59632 > <- bufferStatus - class: > org.apache.http.impl.nio.reactor.IOSessionImpl, value: > org.apache.http.nio.reactor.ssl.SSLIOSession #14910 > <- attachment - class: sun.nio.ch.SelectionKeyImpl, value: > org.apache.http.impl.nio.reactor.IOSessionImpl #14910 > <- value - class: java.util.HashMap$Entry, value: > sun.nio.ch.SelectionKeyImpl #14910 > <- [355] - class: java.util.HashMap$Entry[], value: > java.util.HashMap$Entry #197065 > <- table - class: java.util.HashMap, value: > java.util.HashMap$Entry[] #4612 > <- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl, > value: java.util.HashMap #10016 > > this - value: byte[] #60139 > <- key - class: > com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey, value: > byte[] #60139 > <- masterSecret - class: sun.security.ssl.SSLSessionImpl, value: > com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey #4827 > <- sess - class: sun.security.ssl.SSLEngineImpl, value: > sun.security.ssl.SSLSessionImpl #4829 > <- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession, > value: sun.security.ssl.SSLEngineImpl #20572 > <- bufferStatus - class: > org.apache.http.impl.nio.reactor.IOSessionImpl, value: > org.apache.http.nio.reactor.ssl.SSLIOSession #20572 > <- attachment - class: sun.nio.ch.SelectionKeyImpl, value: > org.apache.http.impl.nio.reactor.IOSessionImpl #20572 > <- value - class: java.util.HashMap$Entry, value: > sun.nio.ch.SelectionKeyImpl #20572 > <- [1182] - class: java.util.HashMap$Entry[], value: > java.util.HashMap$Entry #254234 > <- table - class: java.util.HashMap, value: > java.util.HashMap$Entry[] #6977 > <- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl, > value: java.util.HashMap #9595 > > this - value: byte[] #71913 > <- iv - class: com.sun.crypto.provider.CipherBlockChaining, value: > byte[] #71913 > <- cipher - class: com.sun.crypto.provider.CipherCore, value: > com.sun.crypto.provider.CipherBlockChaining #3709 > <- core - class: com.sun.crypto.provider.AESCipher$General, value: > com.sun.crypto.provider.CipherCore #3709 > <- spi - class: javax.crypto.Cipher, value: > com.sun.crypto.provider.AESCipher$General #3709 > <- cipher - class: sun.security.ssl.CipherBox, value: > javax.crypto.Cipher #3709 > <- readCipher - class: sun.security.ssl.SSLEngineImpl, value: > sun.security.ssl.CipherBox #3710 > <- sslEngine - class: > org.apache.http.nio.reactor.ssl.SSLIOSession, value: > sun.security.ssl.SSLEngineImpl #1855 > <- bufferStatus - class: > org.apache.http.impl.nio.reactor.IOSessionImpl, value: > org.apache.http.nio.reactor.ssl.SSLIOSession #1855 > <- key - class: java.util.HashMap$Entry, value: > org.apache.http.impl.nio.reactor.IOSessionImpl #1855 > <- [4000] - class: java.util.HashMap$Entry[], value: > java.util.HashMap$Entry #71715 > <- table - class: java.util.HashMap, value: > java.util.HashMap$Entry[] #24138 > <- map - class: java.util.HashSet, value: java.util.HashMap > #8449 > <- c - class: java.util.Collections$SynchronizedSet, value: > java.util.HashSet #3312 > <- sessions (Java frame) - class: > org.apache.http.impl.nio.reactor.BaseIOReactor, value: > java.util.Collections$SynchronizedSet #6 > > I can give more details about the configuration of the HTTP Client and the > SSL sessions if it would be helpful, but I don’t want to give too much info > if this is expected behavior. > > Any advice would be appreciated. > > Thanks, > > Hal > --------------------------------------------------------------------- To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org