On Thu, Nov 12, 2020 at 12:34 AM Jerry Malcolm <techst...@malcolms.com> wrote:
> On 11/9/2020 11:05 AM, Jerry Malcolm wrote: > > > > On 11/9/2020 3:10 AM, Mark Thomas wrote: > >> On 08/11/2020 01:33, Jerry Malcolm wrote: > >>> On 11/7/2020 6:56 PM, Christopher Schultz wrote: > >>>> Jerry, > >>>> > >>>> On 11/6/20 19:49, Jerry Malcolm wrote: > >>>>> I have a relatively new environment with a standalone tomcat (8.5) > >>>>> running on an AWS Linux2 EC2. I'm not using HTTPD/AJP. Its a direct > >>>>> connection to port 443. (Well technically, I have firewallD in the > >>>>> flow in order to route the protected port 80 to port 8080 and 443 to > >>>>> 8443 for TC). > >>>>> > >>>>> I am doing some stress testing on the server and failing miserably. > >>>>> I am sending around 130 ajax calls in rapid succession using HTTP/2. > >>>>> These are all very simple small page (JSP) requests. Not a lot of > >>>>> processing required. The first ~75 requests process normally. Then > >>>>> everything hangs up. In the tomcat logs I'm getting a bunch of > >>>>> "Timeout waiting to read data from client" exceptions. And in the > >>>>> stacktrace for these exceptions, they are all occurring when I'm > >>>>> trying to access a parameter from the request. Looking at the > >>>>> request network timing in the browser console, I see a bunch of the > >>>>> requests returning in typical time of a few milliseconds. Then > >>>>> another large block of requests that all start returning around 4 > >>>>> seconds, then another block that wait until 8 seconds to return. > >>>>> I've tried firefox and chrome with the same results. > >>>>> > >>>>> I've been using httpd in front of TC for years. So this is the first > >>>>> time I'm running TC standalone. It is very likely I've got some > >>>>> parameters set horribly wrong. But I have no clue where to start. > >>>>> This is not a tiny EC2, and my internet connection is not showing any > >>>>> signs of problems. So I really don't think this is a > >>>>> performance-related problem. The problem is very consistent and > >>>>> reproducible with the same counts of success/failure calls. What > >>>>> could be causing the "Timeout waiting to read data from client" after > >>>>> 75 calls, and then cause blocks of calls to wait 4 seconds, 8 > >>>>> seconds, etc before responding? I really need to handle more > >>>>> simultaneous load that this is currently allowing. > >>>>> > >>>>> Thanks in advance for the education. > >>>> Are you using HTTP Keepalives on your connections? Are you actually > >>>> re-using those connections in your test? What is your keepalive > >>>> timeout on your <Connector>. Actually, what is your whole <Connector> > >>>> configuration? > >>>> > >>>> -chris > >>>> > >>> Hi Chris, here are my two connector definitions from server.xml: > >>> > >>> <Connector > >>> port="8080" > >>> protocol="HTTP/1.1" > >>> connectionTimeout="20000" > >>> redirectPort="443" /> > >>> > >>> <Connector > >>> port="8443" > >>> maxThreads="150" > >>> connectionTimeout="20000" > >>> SSLEnabled="true" > >>> scheme="https" > >>> secure="true" > >>> clientAuth="false" > >>> SSLCertificateFile="ssl/aaaaa.com/cert.pem" > >>> SSLCertificateChainFile="ssl/aaaaa.com/chain.pem" > >>> SSLCertificateKeyFile="ssl/aaaaa.com/privkey.pem"> > >>> <UpgradeProtocol > >>> className="org.apache.coyote.http2.Http2Protocol" /> > >>> </Connector> > >> How are you stress testing this? All on a single HTTP/2 connection or > >> multiple connections? With which tool? > >> > >> You might want to test HTTP/1.1 requests (with and without TLS) to see > >> if the problem is specific to HTTP/2 or TLS as that should help narrow > >> down the root cause. > >> > >> Mark > > > > Hi Mark, technically it's not a 'designed' stress test. It's real > > production code that just happens to stress the server more than > > usual. It's just a page that makes a bunch of ajax calls, and the > > responses to each of those issue a second ajax call. > > > > If you don't see anything obvious in my configuration, we will > > definitely pursue the http/1.1 options, etc. I just wanted to > > eliminate the chance of obvious 'pilot error' before digging deeper. > > > > Specifically, where is that error detected in the TC flow? In my logs > > it fails on getting request parameters. It sounds like the input > > reader for the request is getting blocked. But the first part of > > the request is getting in since it does route to the appropriate JSP. > > Just seems strange that the http/2 or ssl layers would let half of the > > request in and then block the rest of the request. The browser > > appears to be sending everything. And it fails the same using firefox > > or chrome. Any ideas? > > > > Thx > > > > > Update on this. One of our clients got ERR_HTTP2_SERVER_REFUSED_STREAM > after things locked up. I removed the http2 'upgrade protocol' line > from my connector, and everything works. So it's apparently something > wrong with my http2 setup. Ideas? (See my connector config above in > this thread). > Which version exactly do you use ? There were several big improvements to HTTP2 in the last few releases. > > Thx > >
