Arshiya, On 10/14/20 01:23, Arshiya Shariff wrote: > Please find the answers in-line Mark. > > Http2 requests with message payload of 34KB are pumped from JMeter > at 20 TPS with 700 connections to an application with Embedded tomcat > - 9.0.39 (max-Threads : 200, all other values are the tomcat > defaults) > >> What does that URL do with the POSTed content? Ignore it? Read it >> from an InputStream? Read it via getParameter()? > > The posted content is read via BufferedReader reader = > request.getReader() and processed asynchronously How... exactly?
> Is JMeter run on the same machine as Tomcat? > JMeter is run from a different machine. > > Do you use the JMeter GUI or the command line? > Launched via Command line (JMeter heap increased to 10 GB ) > > What are the specs of the server(s) being used? > The server is a VM with 12 CPUs and 120 GB RAM > > Please let us know if you require more details. This would probabyl be easier if you'd just provide a test-case: a sample (simple!) web application which reproduces what you are reporting. -chris > -----Original Message----- > From: Mark Thomas <ma...@apache.org> > Sent: Monday, October 12, 2020 7:28 PM > To: users@tomcat.apache.org > Subject: Re: HTTP2: memory filled up fast on increasing the connections to > 1000/2000 (Embedded tomcat 9.0.38) > > On 12/10/2020 08:02, Arshiya Shariff wrote: >> Hi Mark , >> >> The issue is reproduced with version 9.0.39 as well. Max threads in Tomcat >> is 200. >> >> Please find the case: >> Client:JMeter 5.2.1 (With http2 plugin) >> TPS: around 20 >> No of users from JMeter : 700 >> Message payload size: 6 KB to 34 KB >> Loop: Infinite >> We let the loop run infinitely and see the java.lang.StackOverflowError >> trace printed multiple times in the log within few minutes of starting the >> test. > > POSTing to what URL? > > What does that URL do with the POSTed content? Ignore it? Read it from an > InputStream? Read it via getParameter()? > > Is JMeter run on the same machine as Tomcat? > > Do you use the JMeter GUI or the command line? > > What are the specs of the server(s) being used? > > You need to provide the exact steps to recreate this issue on a clean install > of Tomcat 9.0.39 as provided by the ASF. > > Mark > > >> Please help us with this . What is the impact of StackOverflowError ? >> >> Thanks and Regards >> Arshiya Shariff >> >> -----Original Message----- >> From: Mark Thomas <ma...@apache.org> >> Sent: Friday, October 9, 2020 5:31 PM >> To: users@tomcat.apache.org >> Subject: Re: HTTP2: memory filled up fast on increasing the >> connections to 1000/2000 (Embedded tomcat 9.0.38) >> >> On 09/10/2020 12:32, Arshiya Shariff wrote: >>> Hi, >>> >>> Mark , with the test runs that I performed over clean 9.0.x branch I was >>> not able to reproduce this. >> >> Good. But I'd really like to understand why... >> >>> But with 9.0.38 and the jars built from 9.0.x with hash: >>> c8ec2d4cde3a31b0e9df9a30e7915d77ba725545 , with 700 or 1000 users >>> (connections) and on sending 1000 Requests per second (or even lesser) , >>> payload of 16K from JMeter I can see that this Exception occurs within few >>> minutes of starting the test . The maxThreads configured in tomcat is 200 . >>> >>> How often do you see these errors in your test run? >>> Randomly, at times 2 or 3 such traces. >> >> OK. Definitely a timing issue then. >> >>> Do you have the other end of that stack trace? >>> It is only the two lines that is recursively printed till the end about >>> ~500 times in one trace : >>> at >>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) >>> at >>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHand >>> l >>> er.completed(SocketWrapperBase.java:1100) >> >> Doesn't tell me much unfortunately. >> >>> I see the trace starting with : >>> Exception in thread "http-nio-x.y.z-1090-exec-107" >>> java.lang.StackOverflowError >>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:446) >>> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) >>> at >>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) >>> at >>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHand >>> l >>> er.completed(SocketWrapperBase.java:1100) >>> >>> (OR) >>> >>> Exception in thread "http-nio-x.y.z-1090-exec-87" >>> java.lang.StackOverflowError >>> at sun.nio.ch.IOVecWrapper.get(IOVecWrapper.java:96) >>> at sun.nio.ch.IOUtil.read(IOUtil.java:240) >>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440) >>> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) >>> at >>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) >>> at >>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) >>> at >>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) >>> at >>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) >>> ..... >>> ..... >>> ..... >>> ..... >>> at >>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) >>> at >>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) >>> at >>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) >>> at >>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHand >>> l >>> er.completed(SocketWrapperBase.java:1100) >>> >>> Is there anything that was fixed around this in latest 9.0.x branch ? >> >> Not obviously. I've reviewed every commit since c8ec2d4c. There is nothing >> that directly works with the I/O. There is 1e97ab2 which fixes a relatively >> recent regression in the HTTP/2 code. I guess it is possible (but it seems a >> bit of a stretch) that that bug is triggering an issue in JMeter which in >> turn is sending invalid HTTP/2 packets. >> >> I think at this point, given the relatively small number of commits between >> c8ec2d4c and HEAD, the most useful thing you could do is run a binary search >> to find out at which commit the issue is fixed. If we know which commit to >> look at that should help track down the root cause. >> >> Mark >> >>> >>> Thanks and Regards >>> Arshiya Shariff >>> >>> -----Original Message----- >>> From: Mark Thomas <ma...@apache.org> >>> Sent: Monday, October 5, 2020 9:52 PM >>> To: users@tomcat.apache.org >>> Subject: Re: HTTP2: memory filled up fast on increasing the >>> connections to 1000/2000 (Embedded tomcat 9.0.38) >>> >>> On 05/10/2020 10:56, Arshiya Shariff wrote: >>>> Hi All, >>>> >>>> Thank you so much Mark . >>>> We tested the jars built from latest 9.0.x with 2000 / 5000 users >>>> (connections) from JMeter , We see a very good improvement with the >>>> heap usage >>> >>> Good news. As is the fact that the other errors have been cleared up. >>> >>>> But I see this exception printed multiple times , I am not sure why this >>>> occurs : >>>> Exception in thread "http-nio-x.y.z-1234-exec-213" >>>> java.lang.StackOverflowError >>>> at sun.nio.ch.IOUtil.read(IOUtil.java:240) >>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440) >>>> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) >>>> at >>>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) >>>> at >>>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) >>>> at >>>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) >>>> at >>>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) >>>> at >>>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) >>>> at >>>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) >>>> at >>>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperation >>>> S >>>> t >>>> ate.run(NioEndpoint.java:1511) >>> >>> That looks like an infinite loop reading an incoming frame. >>> New frames are read using a 9 byte buffer for the header and a 16k buffer >>> for the payload (since Tomcat sets this as the max frame size). >>> >>> The loop is occurring because one of those buffers is simultaneously both >>> full and still has more data to read. That should not be possible and I >>> haven't yet been able to figure out how this is happening. >>> >>> How easy is this to reproduce? >>> >>> How often do you see these errors in your test run? >>> >>> Do you have a reliable test case that reproduces this on a clean Tomcat >>> 9.0.x build? If is, can you share the details? >>> >>> Do you have the other end of that stack trace? I'm interested in how the >>> code enters the loop. >>> >>> Thanks, >>> >>> Mark >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
