Chris > -----Original Message----- > From: Christopher Schultz [mailto:ch...@christopherschultz.net] > Sent: Friday, February 03, 2017 11:08 AM > To: Tomcat Users List > Subject: Re: Increasing maxThreads results in more "Connection refused" > errors? > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > John, > > On 2/3/17 11:17 AM, john.e.gr...@wellsfargo.com.INVALID wrote: > > I'm doing some scalability testing of an app in Tomcat 7.0.73. > > > > Some relevant connector config details: maxThreads=80 > > maxKeepAliveRequests=100 keepAliveTimeout=10000 maxConnections > > unspecified (defaults to maxThreads according to the docs) acceptCount > > unspecified (100 according to the docs) clientAuth=true > > Which connector? > > > FWIW, I'm testing two Tomcat instances on the same server. They are > > behind a load balancer. > > Which load-balancer using which protocol? > > > It appears that when the load generator tries to exceed maxThreads, > > the new connection rate goes up quickly and CPU usage shoots up with > > it. > > When you say the "new connection rate" goes up, what exactly do you mean? > When using a load-generator, I would certainly expect the "new connection > rate" to go up. > > > I assume this is because Tomcat is proactively closing idle keep alive > > connections to service new connections. > > No. Tomcat behaves the same under load as when not under load. If you are > using the BIO connector (it sounds like you are), then each connection will > have > to wait for the KeepAlive timeout to expire before servicing another request > from another client. > > This is complicated by the reverse-proxy and so those details are important to > provide. > > > In an effort to keep the CPU in check, I tried increasing maxThreads > > from 80 to 120. > > If the CPU is very busy, allowing more threads to run is counter-productive, > isn't > it? > > > This seemed to work well in a lot of ways. New connection rate didn't > > increase as much, CPU didn't increase as much, there was more > > connection reuse (more requests per connection,) and response times > > didn't deteriorate as much. > > > > Great, right? > > > > Then I noticed a large increase in "Connection refused" errors on the > > load generator. In other words, a higher maxThreads also results in a > > high error rate. The total hits per second from the client's > > perspective is about 60 in both cases. With maxThreads=80, there are > > about 3 connection refused errors per second at that volume. With > > maxThreads=120, there are about 10 connection refused errors per > > second. > > Are you hitting the load-balancer or Tomcat directly during your load-test? > > If you are hitting the load-balancer, then the details of that configuration > are > more important for the client: if the lb will only accept 10 connections, it > doesn't > matter what Tomcat is willing to accep t. > > > I have no idea why this is. Can someone explain this or what I can do > > about it? > > Post more data. We'll get to the bottom of it. > > - -chris
Thanks for asking those good questions. When I checked with the load balancer people, they told me there was a 70 connection limit to each Tomcat server enforced by the load balancer. It took a while to get that limit removed, but now that it's gone, the connection errors completely went away. I think the limit is a good idea and we will probably re-establish a higher one after additional testing. Thanks John
