On 11/11/2020 22:32, Jerry Malcolm wrote: > On 11/9/2020 11:05 AM, Jerry Malcolm wrote: >> >> On 11/9/2020 3:10 AM, Mark Thomas wrote: >>> On 08/11/2020 01:33, Jerry Malcolm wrote: >>>> On 11/7/2020 6:56 PM, Christopher Schultz wrote: >>>>> Jerry, >>>>> >>>>> On 11/6/20 19:49, Jerry Malcolm wrote: >>>>>> I have a relatively new environment with a standalone tomcat (8.5) >>>>>> running on an AWS Linux2 EC2. I'm not using HTTPD/AJP. Its a direct >>>>>> connection to port 443. (Well technically, I have firewallD in the >>>>>> flow in order to route the protected port 80 to port 8080 and 443 to >>>>>> 8443 for TC). >>>>>> >>>>>> I am doing some stress testing on the server and failing miserably. >>>>>> I am sending around 130 ajax calls in rapid succession using HTTP/2. >>>>>> These are all very simple small page (JSP) requests. Not a lot of >>>>>> processing required. The first ~75 requests process normally. Then >>>>>> everything hangs up. In the tomcat logs I'm getting a bunch of >>>>>> "Timeout waiting to read data from client" exceptions. And in the >>>>>> stacktrace for these exceptions, they are all occurring when I'm >>>>>> trying to access a parameter from the request. Looking at the >>>>>> request network timing in the browser console, I see a bunch of the >>>>>> requests returning in typical time of a few milliseconds. Then >>>>>> another large block of requests that all start returning around 4 >>>>>> seconds, then another block that wait until 8 seconds to return. >>>>>> I've tried firefox and chrome with the same results. >>>>>> >>>>>> I've been using httpd in front of TC for years. So this is the first >>>>>> time I'm running TC standalone. It is very likely I've got some >>>>>> parameters set horribly wrong. But I have no clue where to start. >>>>>> This is not a tiny EC2, and my internet connection is not showing any >>>>>> signs of problems. So I really don't think this is a >>>>>> performance-related problem. The problem is very consistent and >>>>>> reproducible with the same counts of success/failure calls. What >>>>>> could be causing the "Timeout waiting to read data from client" after >>>>>> 75 calls, and then cause blocks of calls to wait 4 seconds, 8 >>>>>> seconds, etc before responding? I really need to handle more >>>>>> simultaneous load that this is currently allowing. >>>>>> >>>>>> Thanks in advance for the education. >>>>> Are you using HTTP Keepalives on your connections? Are you actually >>>>> re-using those connections in your test? What is your keepalive >>>>> timeout on your <Connector>. Actually, what is your whole <Connector> >>>>> configuration? >>>>> >>>>> -chris >>>>> >>>> Hi Chris, here are my two connector definitions from server.xml: >>>> >>>> <Connector >>>> port="8080" >>>> protocol="HTTP/1.1" >>>> connectionTimeout="20000" >>>> redirectPort="443" /> >>>> >>>> <Connector >>>> port="8443" >>>> maxThreads="150" >>>> connectionTimeout="20000" >>>> SSLEnabled="true" >>>> scheme="https" >>>> secure="true" >>>> clientAuth="false" >>>> SSLCertificateFile="ssl/aaaaa.com/cert.pem" >>>> SSLCertificateChainFile="ssl/aaaaa.com/chain.pem" >>>> SSLCertificateKeyFile="ssl/aaaaa.com/privkey.pem"> >>>> <UpgradeProtocol >>>> className="org.apache.coyote.http2.Http2Protocol" /> >>>> </Connector> >>> How are you stress testing this? All on a single HTTP/2 connection or >>> multiple connections? With which tool? >>> >>> You might want to test HTTP/1.1 requests (with and without TLS) to see >>> if the problem is specific to HTTP/2 or TLS as that should help narrow >>> down the root cause. >>> >>> Mark >> >> Hi Mark, technically it's not a 'designed' stress test. It's real >> production code that just happens to stress the server more than >> usual. It's just a page that makes a bunch of ajax calls, and the >> responses to each of those issue a second ajax call. >> >> If you don't see anything obvious in my configuration, we will >> definitely pursue the http/1.1 options, etc. I just wanted to >> eliminate the chance of obvious 'pilot error' before digging deeper. >> >> Specifically, where is that error detected in the TC flow? In my logs >> it fails on getting request parameters. It sounds like the input >> reader for the request is getting blocked. But the first part of >> the request is getting in since it does route to the appropriate JSP. >> Just seems strange that the http/2 or ssl layers would let half of the >> request in and then block the rest of the request. The browser >> appears to be sending everything. And it fails the same using firefox >> or chrome. Any ideas? >> >> Thx >> >> > Update on this. One of our clients got ERR_HTTP2_SERVER_REFUSED_STREAM > after things locked up. I removed the http2 'upgrade protocol' line > from my connector, and everything works. So it's apparently something > wrong with my http2 setup. Ideas? (See my connector config above in > this thread).
Tomcat only issues that error if the client ignores (in violation of the HTTP/2 spec) the server specified limit on the number of concurrent streams. Take a look at the "maxConcurrentStreams" and "maxConcurrentStreamExecution" attributes for Tomcat's HTTP/2 configuration: http://tomcat.apache.org/tomcat-9.0-doc/config/http2.html Mark --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
