On Thu, Mar 7, 2019 at 1:47 PM Mark Thomas <ma...@apache.org> wrote: > On 07/03/2019 07:40, Santhosh Kumar wrote: > > From some of the test cases I can safely say that tomcat is hitting some > > limits, I have two test cases ran with two diff size of payload and > without > > any queryParams. The servlet is a empty servlet just returns after > > receiving without doing any business side logic > > Can you repeat those tests with the NIO connector? It would be helpful > to know if we should be looking at the HTTP/2 code or the low-level > connector I/O code. >
I was planning to investigate since I'm hunting NIO2 additional issues after the fix for BZ63182. This one looks simpler to reproduce at least [assuming there's an issue]. Rémy > > Thanks, > > Mark > > > > > > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d > > /home/local/santhosh/A-Test/nghttp2/agentRequest.txt > > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet > > starting benchmark... > > spawning thread #0: 1 total client(s). 100 total requests > > TLS Protocol: TLSv1.3 > > Cipher: TLS_AES_256_GCM_SHA384 > > Server Temp Key: X25519 253 bits > > Application protocol: h2 > > progress: 10% done > > progress: 20% done > > progress: 30% done > > progress: 40% done > > progress: 50% done > > > > finished in 5.16s, 10.48 req/s, 552B/s > > requests: 100 total, 55 started, 54 done, 54 succeeded, 46 failed, 46 > > errored, 0 timeout > > status codes: 55 2xx, 0 3xx, 0 4xx, 0 5xx > > traffic: 2.78KB (2846) total, 1.77KB (1815) headers (space savings > 43.10%), > > 0B (0) data > > min max mean sd +/- > sd > > time for request: 1.57ms 9.43ms 2.24ms 1.17ms > 94.44% > > time for connect: 4.69ms 4.69ms 4.69ms 0us > 100.00% > > time to 1st byte: 0us 0us 0us 0us > 0.00% > > req/s : 10.48 10.48 10.48 0.00 > 100.00% > > > > This above configuration always returns 54 succeeded, payload size is > 1200B > > (1200x54=64800) > > > ------------------------------------------------------------------------------------------------------------------------------ > > Now reduce the payload and trying the same test, > > > > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d > > /home/local/santhosh/A-Test/nghttp2/agentRequest2.txt > > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet > > starting benchmark... > > spawning thread #0: 1 total client(s). 100 total requests > > TLS Protocol: TLSv1.3 > > Cipher: TLS_AES_256_GCM_SHA384 > > Server Temp Key: X25519 253 bits > > Application protocol: h2 > > progress: 10% done > > progress: 20% done > > progress: 30% done > > progress: 40% done > > progress: 50% done > > progress: 60% done > > progress: 70% done > > progress: 80% done > > > > finished in 5.21s, 16.11 req/s, 839B/s > > requests: 100 total, 85 started, 84 done, 84 succeeded, 16 failed, 16 > > errored, 0 timeout > > status codes: 85 2xx, 0 3xx, 0 4xx, 0 5xx > > traffic: 4.27KB (4376) total, 2.74KB (2805) headers (space savings > 43.10%), > > 0B (0) data > > min max mean sd +/- > sd > > time for request: 1.43ms 5.80ms 2.04ms 760us > 89.29% > > time for connect: 5.02ms 5.02ms 5.02ms 0us > 100.00% > > time to 1st byte: 0us 0us 0us 0us > 0.00% > > req/s : 16.11 16.11 16.11 0.00 > 100.00% > > > > This above configuration always returns 84 succeeded, payload size is > 775B > > (775x84=65100) > > > ------------------------------------------------------------------------------------------------------------------------------ > > Reducing the payload much smaller, > > > > h2load -n200 -c1 -m1 --header="Content-Type:application/json" -d > > /home/local/santhosh/A-Test/nghttp2/agentRequest3.txt > > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet > > starting benchmark... > > spawning thread #0: 1 total client(s). 200 total requests > > TLS Protocol: TLSv1.3 > > Cipher: TLS_AES_256_GCM_SHA384 > > Server Temp Key: X25519 253 bits > > Application protocol: h2 > > progress: 10% done > > progress: 20% done > > progress: 30% done > > progress: 40% done > > progress: 50% done > > progress: 60% done > > progress: 70% done > > progress: 80% done > > progress: 90% done > > > > finished in 5.41s, 34.40 req/s, 1.73KB/s > > requests: 200 total, 187 started, 186 done, 186 succeeded, 14 failed, 14 > > errored, 0 timeout > > status codes: 187 2xx, 0 3xx, 0 4xx, 0 5xx > > traffic: 9.35KB (9578) total, 6.03KB (6171) headers (space savings > 43.10%), > > 0B (0) data > > min max mean sd +/- > sd > > time for request: 1.18ms 13.49ms 1.91ms 1.13ms > 95.16% > > time for connect: 5.93ms 5.93ms 5.93ms 0us > 100.00% > > time to 1st byte: 0us 0us 0us 0us > 0.00% > > req/s : 34.41 34.41 34.41 0.00 > 100.00% > > > > This above configuration always returns 186 succeeded, payload size is > > 356 (356x186=66216) > > > > On Wed, Mar 6, 2019 at 9:15 PM John Dale <jcdw...@gmail.com> wrote: > > > >> When you run your test(s), does it fail after a certain period of > >> time, or just keep on going under a certain number of requests? > >> > >> Also, to confirm: you're sending 1000 Byte + query strings? > >> > >> Are you doing anything in the server side component to verify that > >> your parameters have been received successfully? > >> > >> I seems very possible that there is increased overhead parsing the > >> request (POST) body. That's why I was wondering about the dynamics of > >> your test case. If you can achieve a steady load state, either some > >> optimization of the POST request parser could be done, or you could > >> accept that overhead if it is comparable to other solutions. > >> > >> On 3/6/19, Santhosh Kumar <santhosh...@gmail.com> wrote: > >>> I hope so, I used updated packages/components at the time of > development. > >>> few may be outdated like tomcat native as I was using 1.2.18 while > >>> developing but 1.2.21 got released recently. > >>> > >>> On Wed, Mar 6, 2019 at 6:18 PM John Dale <jcdw...@gmail.com> wrote: > >>> > >>>> Have you upgraded to the most recent release of your major version? > >>>> > >>>> If so, and if this issue still persists, it is something that the core > >>>> development team might want to look at assuming they can replicate the > >>>> issue. > >>>> > >>>> On 3/5/19, Santhosh Kumar <santhosh...@gmail.com> wrote: > >>>>> Sometimes more than 10x > >>>>> > >>>>> On Tue, Mar 5, 2019 at 10:00 PM John Dale <jcdw...@gmail.com> wrote: > >>>>> > >>>>>> How many orders of magnitude slower are the post requests? > >>>>>> > >>>>>> On 3/5/19, Santhosh Kumar <santhosh...@gmail.com> wrote: > >>>>>>> I was testing in the localhost > >>>>>>> > >>>>>>> On Tue, Mar 5, 2019 at 9:32 PM John Dale <jcdw...@gmail.com> > >> wrote: > >>>>>>> > >>>>>>>> Are you running your test client (h2load) on the same machine, > >> same > >>>>>>>> network, or is it over the net (so, like 20ms latency on each > >>>>>>>> request)? The reason I ask is that if you are local (especially), > >>>>>>>> it > >>>>>>>> may queue up too many requests for tomcat to handle in the testing > >>>>>>>> period with its thread pool. Will let you know if I have any > >> other > >>>>>>>> ideas. > >>>>>>>> > >>>>>>>> On 3/5/19, Santhosh Kumar <santhosh...@gmail.com> wrote: > >>>>>>>>> Bytes > >>>>>>>>> > >>>>>>>>> On Tue, Mar 5, 2019 at 9:28 PM John Dale <jcdw...@gmail.com> > >>>> wrote: > >>>>>>>>> > >>>>>>>>>> 1000-1500 MB or KB? > >>>>>>>>>> > >>>>>>>>>> On 3/4/19, Santhosh Kumar <santhosh...@gmail.com> wrote: > >>>>>>>>>>> As per the documentation, > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig > >>>>>>>>>>> > >>>>>>>>>>> this connector supports maxPostSize, by default the limit is > >>>>>>>>>>> set > >>>>>>>>>>> to > >>>>>>>> 2MB > >>>>>>>>>>> > >>>>>>>>>>> On Tue, Mar 5, 2019 at 5:09 AM John Dale <jcdw...@gmail.com> > >>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Does anyone know if this connector supports maxPostSize > >>>>>>>>>>>> parameter? > >>>>>>>>>>>> > >>>>>>>>>>>> On 3/4/19, Santhosh Kumar <santhosh...@gmail.com> wrote: > >>>>>>>>>>>>> Hi, > >>>>>>>>>>>>> > >>>>>>>>>>>>> We have a tomcat instance which is http2 enabled and it > >>>>>>>>>>>>> needs > >>>>>>>>>>>>> to > >>>>>>>>>>>>> serve > >>>>>>>>>>>>> large number of requests using multiplexing, so we have > >>>>>>>>>>>>> configured > >>>>>>>>>>>>> our > >>>>>>>>>>>>> instance as follows, > >>>>>>>>>>>>> > >>>>>>>>>>>>> <Connector port="9191" URIEncoding="UTF-8" > >>>>>>>>>>>>> sslImplementationName="org.apache.tomcat.util.net > >>>>>>>>>>>> .openssl.OpenSSLImplementation" > >>>>>>>>>>>>> protocol="org.apache.coyote.http11.Http11Nio2Protocol" > >>>>>>>>>>>>> maxThreads="50000" > >>>>>>>>>>>>> SSLEnabled="true" > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml" > >>>>>>>>>>>>> compression="on" > >>>> minSpareThreads="25" > >>>>>>>>>>>>> noCompressionUserAgents="gozilla, traviata" scheme="https" > >>>>>>>>>>>>> secure="true" > >>>>>>>>>>>>> keystoreFile="conf/myfile.keystore" > >> keystorePass="password" > >>>>>>>>>>>>> socket.appReadBufSize="81920" > >>>>>>>>>>>>> socket.appWriteBufSize="81920" socket.rxBufSize="251880" > >>>>>>>>>>>>> socket.txBufSize="438000"> > >>>>>>>>>>>>> <UpgradeProtocol compression="on" > >>>>>>>>>>>>> > >>>>>> maxConcurrentStreamExecution="200" > >>>>>>>>>>>>> maxConcurrentStreams="200" > >>>>>>>>>>>>> className="org.apache.coyote.http2.Http2Protocol"/> > >>>>>>>>>>>>> </Connector> > >>>>>>>>>>>>> > >>>>>>>>>>>>> This instance mainly serves concurrent POST request which > >>>> will > >>>>>>>>>>>>> have > >>>>>>>>>>>> payload > >>>>>>>>>>>>> of size, approx 1000-1500, which can be verified by tomcat > >>>>>>>>>>>>> logs > >>>>>>>>>>>>> > >>>>>>>>>>>>> org.apache.coyote.http2.Http2Parser.validateFrame > >>>>>>>>>>>>> Connection > >>>>>> [0], > >>>>>>>>>>>>> Stream > >>>>>>>>>>>>> [19], Frame type [DATA], Flags [1], Payload size [*1195*] > >>>>>>>>>>>>> > >>>>>>>>>>>>> We tested our server with the help of h2load as follows, > >>>>>>>>>>>>> > >>>>>>>>>>>>> h2load -n100 -c1 -m100 https://localhost:9191/ -d > >>>>>>>>>>>>> '/agentRequest.txt' > >>>>>>>>>>>>> > >>>>>>>>>>>>> We are getting this error as follows, > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch > >>>>>>>>>>>>> Connection > >>>>>>>>>>>> [0] > >>>>>>>>>>>>> java.io.IOException: Unable to unwrap data, invalid > >> status > >>>>>>>>>>>>> [BUFFER_OVERFLOW] > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.tomcat.util.net > >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.tomcat.util.net > >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>> java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >> java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> java.base/sun.nio.ch > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> java.base/sun.nio.ch > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> java.base/sun.nio.ch > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.tomcat.util.net > >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.tomcat.util.net > >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.tomcat.util.net > >>>>>>>>>>>> .SecureNio2Channel.read(SecureNio2Channel.java:1067) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.tomcat.util.net > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.tomcat.util.net > >>>>>>>>>>>> .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.tomcat.util.net > >>>>>>>>>>>> .SocketWrapperBase.read(SocketWrapperBase.java:1012) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>> org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.tomcat.util.net > >>>>>>>>>>>> .Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.tomcat.util.net > >>>>>>>>>>>> .SocketProcessorBase.run(SocketProcessorBase.java:49) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.tomcat.util.net > >>>>>>>>>>>> .AbstractEndpoint.processSocket(AbstractEndpoint.java:1048) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.tomcat.util.net > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> org.apache.tomcat.util.net > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:109) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>> java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >> java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) > >>>>>>>>>>>>> > >>>>>>>>>>>>> Why is this error is thrown? How can I configure tomcat to > >>>>>> handle > >>>>>>>>>>>>> concurrent POST requests which have a decent payload? > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> We have tried with various java clients like > >>>>>>>>>>>>> http-client-5-beta, > >>>>>>>>>>>>> jetty > >>>>>>>>>>>>> or > >>>>>>>>>>>>> okhttp3 and spam requests to our tomcat using http2 > >>>>>>>>>>>>> multiplexing > >>>>>>>> and > >>>>>>>>>> we > >>>>>>>>>>>>> found the time taken to process a requests > >>>> increases(sometimes > >>>>>>>>>>>>> even > >>>>>>>>>>>>> 10x) > >>>>>>>>>>>>> when load is increased. > >>>>>>>>>>>>> We have tweaked all common configuration related to http2 > >>>>>>>>>>>>> on > >>>>>> both > >>>>>>>>>>>>> client > >>>>>>>>>>>>> and server side with no luck. > >>>>>>>>>>>>> > >>>>>>>>>>>>> But same tomcat configuration can handle 10s of 1000s of > >>>>>>>>>>>>> get > >>>>>>>> request > >>>>>>>>>>>>> concurrently without a problem, its only creating problem > >>>> with > >>>>>>>>>>>>> POST > >>>>>>>>>>>>> requests. > >>>>>>>>>>>>> > >>>>>>>>>>>>> What is wrong in our configuration? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Kindly someone shed some light. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Tomcat - 9.0.16 > >>>>>>>>>>>>> APR-1.2.18 > >>>>>>>>>>>>> OpenSSL-1.1.1a > >>>>>>>>>>>>> JDK-10.0.2 > >>>>>>>>>>>>> OS - Ubuntu/Centos > >>>>>>>>>>>>> HeapSize - 4GB > >>>>>>>>>>>>> RAM -16GB > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Kindly help > >>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> *With Regards,* > >>>>>>>>>>>>> *Santhosh Kumar J* > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>> > --------------------------------------------------------------------- > >>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>>>>>>>>>> For additional commands, e-mail: > >> users-h...@tomcat.apache.org > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> *With Regards,* > >>>>>>>>>>> *Santhosh Kumar J* > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>> --------------------------------------------------------------------- > >>>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>>>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>> *With Regards,* > >>>>>>>>> *Santhosh Kumar J* > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >> --------------------------------------------------------------------- > >>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>>>>>>> > >>>>>>>> -- > >>>>>>> *With Regards,* > >>>>>>> *Santhosh Kumar J* > >>>>>>> > >>>>>> > >>>>>> > --------------------------------------------------------------------- > >>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>>>>> > >>>>>> > >>>>> > >>>>> -- > >>>>> *With Regards,* > >>>>> *Santhosh Kumar J* > >>>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>>> > >>>> > >>> > >>> -- > >>> *With Regards,* > >>> *Santhosh Kumar J* > >>> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
