On Thu, 2023-05-11 at 19:17 +0200, Joan grupoventus wrote: > Hello Oleg, > > We are finding an issue in a new installation of our app that is > using httpclient5-5.2.1 and httpcore5-5.2. > > Our app is a proxy that receives protobuf requests that are sent to > an amazon endpoint that is responding with protobuf responses. We > started this communication by using plain http, processing around > 4.000 req/s. Everything OK. > > But when we move to live https must be used to communicate with this > endpoint, just changing the url from http:// to https:// > > So we changed it and after 30 seconds testing we got an > OutOfMemoryException, the heap was exhausted. > > Below an histogram using http (just the first 2 items): > num #instances #bytes class name > ---------------------------------------------- > 1: 181753 52412760 <methodKlass> > 2: 218819 46398592 [B > > > And the histogram using https after 30s: > num #instances #bytes class name > ---------------------------------------------- > 1: 224873 3216328416 [B > 2: 182417 52604088 <methodKlass> > > > So the space occupied by byte arrays moves from 46Mb to 3GB. So we > decided to perform a heap dump and analyze it to find out where these > 3GB byte arrays are coming from, and we have seen this: > > 178 instances of > "org.apache.hc.core5.http2.impl.nio.ClientH2StreamMultiplexer", > loaded by "org.apache.catalina.loader.ParallelWebappClassLoader @ > 0x400768258e8" occupy 2.999.171.840 (88,20 %) bytes > http://www.grupoventus.com/resources/top_dominator_class.png > > > If we analyze each one of these 178 instances of > ClientH2StreamMultiplexer, it seems the OutputBuffer is holding > around 16Mb for each instance, multiplied by 178 gives as a result > these 3GB: > http://www.grupoventus.com/resources/output_buffer.png > > > I’m not sure if this is a bug, we have been using https with these > same versions against other endpoints and we never had a problem. > Anyways it does not seem a memory leak, because if you keep the > traffic stable, the space occupied by these instances seems to not be > growing. > > Let me know what you think, and if you need more information (logs > about http traffic, etc). > > Thanks, > > Joan. >
Hi Joan This sounds odd. I see no reason why FrameOutputBuffer should ever grow beyond the size of the maximum frame size and do not see how this could be possibly be related to TLS. Nevertheless, this may well be a bug that gets triggered by the use of the transport encryption. Please do one thing, though, first. Please make sure you configure the I/O reactors to use a custom TLS strategy with SSLBufferMode set to DYNAMIC. This will make the I/O reactor release intermediate TLS buffers which should substantially decrease the total memory footprint of TLS connections at the cost of extra memory allocation / de- allocation. Oleg --------------------------------------------------------------------- To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
