On Fri, 2023-05-12 at 14:17 +0100, thc...@gmail.com wrote:
>
>
> On 12/05/2023 13:27, Oleg Kalnichevski wrote:
> > On Fri, 2023-05-12 at 11:48 +0200, Joan grupoventus wrote:
> > > Hi Oleg,
> > >
> > > After changing this method from:
> > > private TlsStrategy getTLSStrategy() throws Exception {
> > > return (DefaultClientTlsStrategy.getDefault());
> > > }
> > >
> > > To:
> > > private TlsStrategy getTLSStrategy() throws Exception {
> > > return
> > > ClientTlsStrategyBuilder.create().setSslBufferMode(SSLBufferMode.
> > > DYNA
> > > MIC).build();
> > > }
> > >
> > > And set this strategy to the pool:
> > > this.phccm =
> > > PoolingAsyncClientConnectionManagerBuilder.create().setPoolConcur
> > > renc
> > > yPolicy(PoolConcurrencyPolicy.LAX).setTlsStrategy(this.getTLSStra
> > > tegy
> > > ()).build();
> > >
> > >
> > > The histogram shows the same:
> > > num #instances #bytes class name
> > > ----------------------------------------------
> > > 1: 233151 3208489352 [B
> > > 2: 182882 52757328 <methodKlass>
> > >
> > >
> > > This chart shows the execution under http and the change to https
> > > at
> > > 11:23:35:
> > > http://www.grupoventus.com/resources/spike.png
> > >
> > > With http, at the end of each GC cycle the amount of heap used is
> > > around 1,5GB.
> > > At the moment we dynamically change to https you can see a spike
> > > from
> > > 3GB to 7GB. Next GC cycles frees memory until around 5GB, and
> > > remains
> > > there.
> > >
> > > Here the chart with the execution already with https during
> > > several
> > > minutes, never lows from the 5GB:
> > > http://www.grupoventus.com/resources/https.png
> > >
> > > The point is that here traffic is very low, in production this
> > > traffic is much higher so the spike is about 20GB and the heap
> > > becomes exhausted.
> > >
> > > Joan.
> > >
> >
> > This unfortunately does not help. This is only so much I can do
> > just by
> > looking at the charts.
> >
> > There are two options going forward:
> >
> > 1. You manage to reproduce the issue in an isolated environment
> > that I
> > can replicate locally (Docker image or a unit test).
> >
> > 2. You manage to find out the cause of FrameOutputBuffer growing
> > past
> > the max frame length in your local environment and propose a fix,
> > which
> > I can review and test locally.
>
> fwiw, I don't think it's growing past the max frame length, it's at
> the
> max (16MB).
>
> I see this as well and my conclusion (which might be wrong) is that
> HC
> allocates the max immediately and it's kept that way for as long as
> the
> connection is open (which for HTTP/2 might be a long time).
> So the more connections you have open the more memory is allocated
> (and
> for my case actually unused/needed, in my case it's not sending nor
> receiving more than 100s of KBs).
>
> afaik to reproduce you just need a server advertising that it can
> handle
> the max frame length and HC will reserve the whole chunk. (I also
> tried
> using a custom H2Config with smaller max frame length but it still
> allocates what the other endpoint advertises.)
>
> Hopefully this helps somehow.
>
> Best regards.
>
Please configure HttpClient to use smaller (much smaller) max frame
length.
Oleg
> >
> > Oleg
> >
> >
> > >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski <ol...@apache.org>
> > > Sent: Friday, May 12, 2023 9:47 AM
> > > To: HttpClient User Discussion <httpclient-users@hc.apache.org>
> > > Subject: Re: Httpclient issue with https
> > > "org.apache.hc.core5.http2.impl.nio.ClientH2StreamMultiplexer"
> > >
> > > On Thu, 2023-05-11 at 19:17 +0200, Joan grupoventus wrote:
> > > > Hello Oleg,
> > > >
> > > > We are finding an issue in a new installation of our app that
> > > > is
> > > > using
> > > > httpclient5-5.2.1 and httpcore5-5.2.
> > > >
> > > > Our app is a proxy that receives protobuf requests that are
> > > > sent to
> > > > an
> > > > amazon endpoint that is responding with protobuf responses. We
> > > > started
> > > > this communication by using plain http, processing around
> > > > 4.000 req/s. Everything OK.
> > > >
> > > > But when we move to live https must be used to communicate with
> > > > this
> > > > endpoint, just changing the url from http:// to https://
> > > >
> > > > So we changed it and after 30 seconds testing we got an
> > > > OutOfMemoryException, the heap was exhausted.
> > > >
> > > > Below an histogram using http (just the first 2 items):
> > > > num #instances #bytes class name
> > > > ----------------------------------------------
> > > > 1: 181753 52412760 <methodKlass>
> > > > 2: 218819 46398592 [B
> > > >
> > > >
> > > > And the histogram using https after 30s:
> > > > num #instances #bytes class name
> > > > ----------------------------------------------
> > > > 1: 224873 3216328416 [B
> > > > 2: 182417 52604088 <methodKlass>
> > > >
> > > >
> > > > So the space occupied by byte arrays moves from 46Mb to 3GB. So
> > > > we
> > > > decided to perform a heap dump and analyze it to find out where
> > > > these
> > > > 3GB byte arrays are coming from, and we have seen this:
> > > >
> > > > 178 instances of
> > > > "org.apache.hc.core5.http2.impl.nio.ClientH2StreamMultiplexer",
> > > > loaded by "org.apache.catalina.loader.ParallelWebappClassLoader
> > > > @
> > > > 0x400768258e8" occupy 2.999.171.840 (88,20 %) bytes
> > > > http://www.grupoventus.com/resources/top_dominator_class.png
> > > >
> > > >
> > > > If we analyze each one of these 178 instances of
> > > > ClientH2StreamMultiplexer, it seems the OutputBuffer is holding
> > > > around
> > > > 16Mb for each instance, multiplied by 178 gives as a result
> > > > these
> > > > 3GB:
> > > > http://www.grupoventus.com/resources/output_buffer.png
> > > >
> > > >
> > > > I’m not sure if this is a bug, we have been using https with
> > > > these
> > > > same versions against other endpoints and we never had a
> > > > problem.
> > > > Anyways it does not seem a memory leak, because if you keep the
> > > > traffic stable, the space occupied by these instances seems to
> > > > not
> > > > be
> > > > growing.
> > > >
> > > > Let me know what you think, and if you need more information
> > > > (logs
> > > > about http traffic, etc).
> > > >
> > > > Thanks,
> > > >
> > > > Joan.
> > > >
> > >
> > > Hi Joan
> > >
> > > This sounds odd. I see no reason why FrameOutputBuffer should
> > > ever
> > > grow beyond the size of the maximum frame size and do not see how
> > > this could be possibly be related to TLS. Nevertheless, this may
> > > well
> > > be a bug that gets triggered by the use of the transport
> > > encryption.
> > >
> > > Please do one thing, though, first. Please make sure you
> > > configure
> > > the I/O reactors to use a custom TLS strategy with SSLBufferMode
> > > set
> > > to DYNAMIC. This will make the I/O reactor release intermediate
> > > TLS
> > > buffers which should substantially decrease the total memory
> > > footprint of TLS connections at the cost of extra memory
> > > allocation /
> > > de- allocation.
> > >
> > > Oleg
> > >
> > >
> > > -----------------------------------------------------------------
> > > ----
> > > To unsubscribe, e-mail:
> > > httpclient-users-unsubscr...@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-h...@hc.apache.org
> > >
> > >
> > >
> > >
> > > -----------------------------------------------------------------
> > > ----
> > > To unsubscribe, e-mail:
> > > httpclient-users-unsubscr...@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-h...@hc.apache.org
> > >
> >
> >
> > -------------------------------------------------------------------
> > --
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail:
> > httpclient-users-h...@hc.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
