On Thu, 2016-03-24 at 12:26 +0000, Harold Rosenberg wrote:
> Using the ReleasableSSLBufferManagementStrategy gave a nice improvement. For
> the same run with 7500 simulated users, the change reduced the average live
> data in the heap after a full GC from 3.2GB to about 1.5GB. The amount of
> live data is still proportional to the number of connections (about 3GB of
> live data for a run with 15,000 simulated users), but that is to be expected.
>
> Thanks for the help.
>
> Hal
>
I am glad it has helped.
Feel free to raise JIRA with a feature request to make use of custom
SSLBufferManagementStrategy instances less cumbersome.
Oleg
>
>
>
> On 3/23/16, 3:59 PM, "Oleg Kalnichevski" <ol...@apache.org> wrote:
>
> >On Wed, 2016-03-23 at 18:54 +0000, Harold Rosenberg wrote:
> >> I am seeing unexpected behavior in the memory usage of a program that uses
> >> the Async Http Client. The issue arises when I switch from using
> >> exclusively HTTP connections to exclusively HTTPS connections. In the
> >> latter case, there is a large amount of memory used for byte arrays that
> >> seem to be related to SSL processing, and that aren’t ever cleaned up by
> >> garbage collection.
> >>
> >> My question is whether this is expected due to differences between SSL and
> >> non-SSL processing, or whether I am failing to properly clean up my SSL
> >> connections.
> >>
> >> The program is a workload driver for a performance benchmark. It
> >> simulates user activity to a web service. Each simulated user has about
> >> four active connections to the service. Normally the connections are all
> >> over SSL (really TLS), but for testing purposes there is a switch to use
> >> only non-SSL connections. The configuration of the HTTP client is
> >> identical for in both cases.
> >>
> >> I gathered garbage-collection data with and without TLS for runs with
> >> 7,500 simulated users, which would average about 30,000 active
> >> connections. A user would be active for about five minutes, at which
> >> point the they log out and a new user logs in. While the user is logged
> >> in the connections will occasionally be idle and thus closed due to
> >> keepalive timeout. The runs were two hours long and the data was
> >> collected during a steady-state. Data is generated in the heap at about
> >> the same rate in both cases, but when running with HTTPS there is about
> >> 3.2GB of live data left in the heap after a full garbage-collection, while
> >> with just HTTP there is only about 890MB. The amount of live data in the
> >> heap doesn’t grow once it reaches the 3.2GB mark, so it isn’t a memory
> >> leak, but it is definitely proportional to the number of active
> >> connections. When I do the same run with 15,000 users, at steady-state
> >> the live data in the heap after a full GC is about 6.4GB.
> >>
> >> Of the 3.2GB of live data, about 2.8GB is in 876,041 instances of byte[].
> >> Almost all of the instances seem to be related to SSL processing. Here is
> >> the path to the nearest GC root for a few random instances:
> >>
> >
> >Hi Harold,
> >
> >Could you try using ReleasableSSLBufferManagementStrategy instead of
> >default PermanentSSLBufferManagementStrategy?
> >
> >---
> >DefaultHostnameVerifier hostnameVerifier = new
> >DefaultHostnameVerifier(PublicSuffixMatcherLoader.getDefault());
> >Registry<SchemeIOSessionStrategy> registry =
> >RegistryBuilder.<SchemeIOSessionStrategy>create()
> > .register("http", NoopIOSessionStrategy.INSTANCE)
> > .register("https", new SchemeIOSessionStrategy() {
> > @Override
> > public boolean isLayeringRequired() {
> > return true;
> > }
> >
> > @Override
> > public IOSession upgrade(final HttpHost host, final IOSession
> > iosession) throws IOException {
> > final SSLIOSession ssliosession = new SSLIOSession(
> > iosession,
> > SSLMode.CLIENT,
> > host,
> > SSLContexts.createSystemDefault(),
> > new SSLSetupHandler() {
> >
> > @Override
> > public void initalize(final SSLEngine sslengine)
> > throws SSLException {
> > }
> >
> > @Override
> > public void verify(
> > final IOSession iosession,
> > final SSLSession sslsession) throws
> > SSLException {
> > if
> > (!hostnameVerifier.verify(host.getHostName(), sslsession)) {
> > final Certificate[] certs =
> > sslsession.getPeerCertificates();
> > final X509Certificate x509 =
> > (X509Certificate) certs[0];
> > final X500Principal x500Principal =
> > x509.getSubjectX500Principal();
> > throw new
> > SSLPeerUnverifiedException("Host name '" + host.getHostName() + "' does not
> > match " +
> > "the certificate subject
> > provided by the peer (" + x500Principal.toString() + ")");
> > }
> > }
> >
> > },
> > new ReleasableSSLBufferManagementStrategy());
> > iosession.setAttribute(SSLIOSession.SESSION_KEY,
> > ssliosession);
> > ssliosession.initialize();
> > return ssliosession;
> > }
> > })
> > .build();
> >
> >ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor();
> >PoolingNHttpClientConnectionManager cm = new
> >PoolingNHttpClientConnectionManager(ioReactor, registry);
> >CloseableHttpAsyncClient httpclient = HttpAsyncClients.createMinimal(cm);
> >try {
> > httpclient.start();
> > HttpGet request = new
> > HttpGet("https://urldefense.proofpoint.com/v2/url?u=https-3A__verisign.com_&d=BQIDaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=9uYCHvq1QDKcMAIXCt6iT2etdTpbRn0toGoRKE6QxRo&m=HkvNV6bDDfPghsn_MF5SgU7KlMhzmAGjcfgAmviJCSs&s=w1-QtI1DaVx22Oo8UumpY0uvEblsK5t55MihZ_UwmPM&e=
> > ");
> > Future<HttpResponse> future = httpclient.execute(request, null);
> > HttpResponse response = future.get();
> > System.out.println("Response: " + response.getStatusLine());
> > System.out.println("Shutting down");
> >} finally {
> > httpclient.close();
> >}
> >System.out.println("Done");
> >---
> >
> >Oleg
> >
> >> this - value: byte[] #483799
> >> <- hb - class: java.nio.HeapByteBuffer, value: byte[] #483799
> >> <- buffer - class:
> >> org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer,
> >> value: java.nio.HeapByteBuffer #89456
> >> <- outPlain - class: org.apache.http.nio.reactor.ssl.SSLIOSession,
> >> value:
> >> org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer
> >> #59632
> >> <- bufferStatus - class:
> >> org.apache.http.impl.nio.reactor.IOSessionImpl, value:
> >> org.apache.http.nio.reactor.ssl.SSLIOSession #14910
> >> <- attachment - class: sun.nio.ch.SelectionKeyImpl, value:
> >> org.apache.http.impl.nio.reactor.IOSessionImpl #14910
> >> <- value - class: java.util.HashMap$Entry, value:
> >> sun.nio.ch.SelectionKeyImpl #14910
> >> <- [355] - class: java.util.HashMap$Entry[], value:
> >> java.util.HashMap$Entry #197065
> >> <- table - class: java.util.HashMap, value:
> >> java.util.HashMap$Entry[] #4612
> >> <- fdToKey (Java frame) - class:
> >> sun.nio.ch.EPollSelectorImpl, value: java.util.HashMap #10016
> >>
> >> this - value: byte[] #60139
> >> <- key - class:
> >> com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey,
> >> value: byte[] #60139
> >> <- masterSecret - class: sun.security.ssl.SSLSessionImpl, value:
> >> com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey #4827
> >> <- sess - class: sun.security.ssl.SSLEngineImpl, value:
> >> sun.security.ssl.SSLSessionImpl #4829
> >> <- sslEngine - class:
> >> org.apache.http.nio.reactor.ssl.SSLIOSession, value:
> >> sun.security.ssl.SSLEngineImpl #20572
> >> <- bufferStatus - class:
> >> org.apache.http.impl.nio.reactor.IOSessionImpl, value:
> >> org.apache.http.nio.reactor.ssl.SSLIOSession #20572
> >> <- attachment - class: sun.nio.ch.SelectionKeyImpl, value:
> >> org.apache.http.impl.nio.reactor.IOSessionImpl #20572
> >> <- value - class: java.util.HashMap$Entry, value:
> >> sun.nio.ch.SelectionKeyImpl #20572
> >> <- [1182] - class: java.util.HashMap$Entry[], value:
> >> java.util.HashMap$Entry #254234
> >> <- table - class: java.util.HashMap, value:
> >> java.util.HashMap$Entry[] #6977
> >> <- fdToKey (Java frame) - class:
> >> sun.nio.ch.EPollSelectorImpl, value: java.util.HashMap #9595
> >>
> >> this - value: byte[] #71913
> >> <- iv - class: com.sun.crypto.provider.CipherBlockChaining, value:
> >> byte[] #71913
> >> <- cipher - class: com.sun.crypto.provider.CipherCore, value:
> >> com.sun.crypto.provider.CipherBlockChaining #3709
> >> <- core - class: com.sun.crypto.provider.AESCipher$General, value:
> >> com.sun.crypto.provider.CipherCore #3709
> >> <- spi - class: javax.crypto.Cipher, value:
> >> com.sun.crypto.provider.AESCipher$General #3709
> >> <- cipher - class: sun.security.ssl.CipherBox, value:
> >> javax.crypto.Cipher #3709
> >> <- readCipher - class: sun.security.ssl.SSLEngineImpl, value:
> >> sun.security.ssl.CipherBox #3710
> >> <- sslEngine - class:
> >> org.apache.http.nio.reactor.ssl.SSLIOSession, value:
> >> sun.security.ssl.SSLEngineImpl #1855
> >> <- bufferStatus - class:
> >> org.apache.http.impl.nio.reactor.IOSessionImpl, value:
> >> org.apache.http.nio.reactor.ssl.SSLIOSession #1855
> >> <- key - class: java.util.HashMap$Entry, value:
> >> org.apache.http.impl.nio.reactor.IOSessionImpl #1855
> >> <- [4000] - class: java.util.HashMap$Entry[], value:
> >> java.util.HashMap$Entry #71715
> >> <- table - class: java.util.HashMap, value:
> >> java.util.HashMap$Entry[] #24138
> >> <- map - class: java.util.HashSet, value:
> >> java.util.HashMap #8449
> >> <- c - class: java.util.Collections$SynchronizedSet,
> >> value: java.util.HashSet #3312
> >> <- sessions (Java frame) - class:
> >> org.apache.http.impl.nio.reactor.BaseIOReactor, value:
> >> java.util.Collections$SynchronizedSet #6
> >>
> >> I can give more details about the configuration of the HTTP Client and the
> >> SSL sessions if it would be helpful, but I don’t want to give too much
> >> info if this is expected behavior.
> >>
> >> Any advice would be appreciated.
> >>
> >> Thanks,
> >>
> >> Hal
> >>
> >
> >
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> >For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
