I am seeing unexpected behavior in the memory usage of a program that uses the
Async Http Client. The issue arises when I switch from using exclusively HTTP
connections to exclusively HTTPS connections. In the latter case, there is a
large amount of memory used for byte arrays that seem to be related to SSL
processing, and that aren’t ever cleaned up by garbage collection.
My question is whether this is expected due to differences between SSL and
non-SSL processing, or whether I am failing to properly clean up my SSL
connections.
The program is a workload driver for a performance benchmark. It simulates
user activity to a web service. Each simulated user has about four active
connections to the service. Normally the connections are all over SSL (really
TLS), but for testing purposes there is a switch to use only non-SSL
connections. The configuration of the HTTP client is identical for in both
cases.
I gathered garbage-collection data with and without TLS for runs with 7,500
simulated users, which would average about 30,000 active connections. A user
would be active for about five minutes, at which point the they log out and a
new user logs in. While the user is logged in the connections will
occasionally be idle and thus closed due to keepalive timeout. The runs were
two hours long and the data was collected during a steady-state. Data is
generated in the heap at about the same rate in both cases, but when running
with HTTPS there is about 3.2GB of live data left in the heap after a full
garbage-collection, while with just HTTP there is only about 890MB. The amount
of live data in the heap doesn’t grow once it reaches the 3.2GB mark, so it
isn’t a memory leak, but it is definitely proportional to the number of active
connections. When I do the same run with 15,000 users, at steady-state the
live data in the heap after a full GC is about 6.4GB.
Of the 3.2GB of live data, about 2.8GB is in 876,041 instances of byte[].
Almost all of the instances seem to be related to SSL processing. Here is the
path to the nearest GC root for a few random instances:
this - value: byte[] #483799
<- hb - class: java.nio.HeapByteBuffer, value: byte[] #483799
<- buffer - class:
org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer,
value: java.nio.HeapByteBuffer #89456
<- outPlain - class: org.apache.http.nio.reactor.ssl.SSLIOSession,
value:
org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer
#59632
<- bufferStatus - class:
org.apache.http.impl.nio.reactor.IOSessionImpl, value:
org.apache.http.nio.reactor.ssl.SSLIOSession #14910
<- attachment - class: sun.nio.ch.SelectionKeyImpl, value:
org.apache.http.impl.nio.reactor.IOSessionImpl #14910
<- value - class: java.util.HashMap$Entry, value:
sun.nio.ch.SelectionKeyImpl #14910
<- [355] - class: java.util.HashMap$Entry[], value:
java.util.HashMap$Entry #197065
<- table - class: java.util.HashMap, value:
java.util.HashMap$Entry[] #4612
<- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl,
value: java.util.HashMap #10016
this - value: byte[] #60139
<- key - class:
com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey, value:
byte[] #60139
<- masterSecret - class: sun.security.ssl.SSLSessionImpl, value:
com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey #4827
<- sess - class: sun.security.ssl.SSLEngineImpl, value:
sun.security.ssl.SSLSessionImpl #4829
<- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession,
value: sun.security.ssl.SSLEngineImpl #20572
<- bufferStatus - class:
org.apache.http.impl.nio.reactor.IOSessionImpl, value:
org.apache.http.nio.reactor.ssl.SSLIOSession #20572
<- attachment - class: sun.nio.ch.SelectionKeyImpl, value:
org.apache.http.impl.nio.reactor.IOSessionImpl #20572
<- value - class: java.util.HashMap$Entry, value:
sun.nio.ch.SelectionKeyImpl #20572
<- [1182] - class: java.util.HashMap$Entry[], value:
java.util.HashMap$Entry #254234
<- table - class: java.util.HashMap, value:
java.util.HashMap$Entry[] #6977
<- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl,
value: java.util.HashMap #9595
this - value: byte[] #71913
<- iv - class: com.sun.crypto.provider.CipherBlockChaining, value: byte[]
#71913
<- cipher - class: com.sun.crypto.provider.CipherCore, value:
com.sun.crypto.provider.CipherBlockChaining #3709
<- core - class: com.sun.crypto.provider.AESCipher$General, value:
com.sun.crypto.provider.CipherCore #3709
<- spi - class: javax.crypto.Cipher, value:
com.sun.crypto.provider.AESCipher$General #3709
<- cipher - class: sun.security.ssl.CipherBox, value:
javax.crypto.Cipher #3709
<- readCipher - class: sun.security.ssl.SSLEngineImpl, value:
sun.security.ssl.CipherBox #3710
<- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession,
value: sun.security.ssl.SSLEngineImpl #1855
<- bufferStatus - class:
org.apache.http.impl.nio.reactor.IOSessionImpl, value:
org.apache.http.nio.reactor.ssl.SSLIOSession #1855
<- key - class: java.util.HashMap$Entry, value:
org.apache.http.impl.nio.reactor.IOSessionImpl #1855
<- [4000] - class: java.util.HashMap$Entry[], value:
java.util.HashMap$Entry #71715
<- table - class: java.util.HashMap, value:
java.util.HashMap$Entry[] #24138
<- map - class: java.util.HashSet, value: java.util.HashMap
#8449
<- c - class: java.util.Collections$SynchronizedSet, value:
java.util.HashSet #3312
<- sessions (Java frame) - class:
org.apache.http.impl.nio.reactor.BaseIOReactor, value:
java.util.Collections$SynchronizedSet #6
I can give more details about the configuration of the HTTP Client and the SSL
sessions if it would be helpful, but I don’t want to give too much info if this
is expected behavior.
Any advice would be appreciated.
Thanks,
Hal
