On 25/06/2020 11:00, Chirag Dewan wrote: > Thanks for the quick check Mark. > > These are the images I tried referring to: > > https://ibb.co/LzKtRgh > > https://ibb.co/2s7hqRL > > https://ibb.co/KmKj590 > > > The last one is the MAT screenshot showing many RequestInfo objects.
Thanks. That certainly looks like a memory leak. I'll take a closer look. Out of interest, how many threads is the Connector configured to use? Mark > > > Thanks, > > Chirag > > On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote: > >> On 24/06/2020 12:17, Mark Thomas wrote: >>> On 22/06/2020 11:06, Chirag Dewan wrote: >>>> Hi, >>>> >>>> Update: We found that Tomcat goes OOM when a client closes and opens new >>>> connections every second. In the memory dump, we see a lot of >>>> RequestInfo objects that are causing the memory spike. >>>> >>>> After a while, Tomcat goes OOM and start rejecting request(I get a >>>> request timed out on my client). This seems like a bug to me. >>>> >>>> For better understanding, let me explain my use case again: >>>> >>>> I have a jetty client that sends HTTP2 requests to Tomcat. My >>>> requirement is to close a connection after a configurable(say 5000) >>>> number of requests/streams and open a new connection that continues to >>>> send requests. I close a connection by sending a GoAway frame. >>>> >>>> When I execute this use case under load, I see that after ~2hours my >>>> requests fail and I get a series of errors like request >>>> timeouts(5seconds), invalid window update frame, and connection close >>>> exception on my client. >>>> On further debugging, I found that it's a Tomcat memory problem and it >>>> goes OOM after sometime under heavy load with multiple connections being >>>> re-established by the clients. >>>> >>>> image.png >>>> >>>> image.png >>>> >>>> Is this a known issue? Or a known behavior with Tomcat? >>> >>> Embedded images get dropped by the list software. Post those images >>> somewhere we can see them. >>> >>>> Please let me know if you any experience with such a situation. Thanks >>>> in advance. >>> >>> Nothing comes to mind. >>> >>> I'll try some simple tests with HTTP/2. >> >> I don't see a memory leak (the memory is reclaimed eventually) but I do >> see possibilities to release memory associated with request processing >> sooner. >> >> Right now you need to allocate more memory to the Java process to enable >> Tomcat to handle the HTTP/2 load it is presented with. >> >> It looks like a reasonable chunk of memory is released when the >> Connection closes that could be released earlier when the associated >> Stream closes. I'll take a look at what can be done in that area. In the >> meantime, reducing the number of Streams you allow on a Connection >> before it is closed should reduce overall memory usage. >> >> Mark >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
