Re: Increasing maxThreads results in more "Connection refused" errors?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 John, On 2/3/17 11:17 AM, john.e.gr...@wellsfargo.com.INVALID wrote: > I'm doing some scalability testing of an app in Tomcat 7.0.73. > > Some relevant connector config details: maxThreads=80 > maxKeepAliveRequests=100 keepAliveTimeout=1 maxConnections > unspecified (defaults to maxThreads according to the docs) > acceptCount unspecified (100 according to the docs) > clientAuth=true Which connector? > FWIW, I'm testing two Tomcat instances on the same server. They > are behind a load balancer. Which load-balancer using which protocol? > It appears that when the load generator tries to exceed > maxThreads, the new connection rate goes up quickly and CPU usage > shoots up with it. When you say the "new connection rate" goes up, what exactly do you mean? When using a load-generator, I would certainly expect the "new connection rate" to go up. > I assume this is because Tomcat is proactively closing idle keep > alive connections to service new connections. No. Tomcat behaves the same under load as when not under load. If you are using the BIO connector (it sounds like you are), then each connection will have to wait for the KeepAlive timeout to expire before servicing another request from another client. This is complicated by the reverse-proxy and so those details are important to provide. > In an effort to keep the CPU in check, I tried increasing > maxThreads from 80 to 120. If the CPU is very busy, allowing more threads to run is counter-productive, isn't it? > This seemed to work well in a lot of ways. New connection rate > didn't increase as much, CPU didn't increase as much, there was > more connection reuse (more requests per connection,) and response > times didn't deteriorate as much. > > Great, right? > > Then I noticed a large increase in "Connection refused" errors on > the load generator. In other words, a higher maxThreads also > results in a high error rate. The total hits per second from the > client's perspective is about 60 in both cases. With maxThreads=80, > there are about 3 connection refused errors per second at that > volume. With maxThreads=120, there are about 10 connection refused > errors per second. Are you hitting the load-balancer or Tomcat directly during your load-test? If you are hitting the load-balancer, then the details of that configuration are more important for the client: if the lb will only accept 10 connections, it doesn't matter what Tomcat is willing to accep t. > I have no idea why this is. Can someone explain this or what I can > do about it? Post more data. We'll get to the bottom of it. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJYlLjdAAoJEBzwKT+lPKRYRyYQAKs8F2HQ60cmlInZuubIQaq5 MuJxMZV/rCSgFMwoBzR+KDMS2PlTUvpoUMlp3/tTz8oON4x6j3jGzQdA9EGcjje4 W7D/G6TmoMptjeI8Zz0UbYnIRM5JXEvxOzuuGURbER7iCaa9VVsoWnhcv+Ja02Ut Xl9dzEwFpfXasr8ioHL5Yn4W7F0QUDz9vboBQ/9TwYAib9UAKEP8uzWWjh6ylLaw Z9ShyJtCU5rQ47ZVYhCcd+ZgCnP0sbcjD9kloD29dDFHgALTWPuAQA50ZDPiAUBR I0WuDMGITpsl0ni4jbwMjrnv3voaVD0HtSCZon6fynux2BciaHVGx8s0lGtRMz6v isG5L6+io8JR3asNdb8Ug3G+5zX8FMZx5VwlsQNVqo10xsEXYosACtqTmASyLPx6 EwgLDnCeMwKK3A6hmKqw0aEl1R9TqlpLKK5h9g7+RPpm6imUdsZvHAjSL3OWJa8e Laqujf8MxCN/DsZhKOocwv5F98zcsn/mD9VZ/2IG+78JqfB3qU4wa5brD11n71xe uYMgGIH0zIGN9ZGRtWdAbPbY2JIu2qIkFzz06hgYr/W5QLrZzy2dGYkyGBPMHrDV TPR1+EbNNOtGVq5IF6t8ZChOE6toNnIdWdy5bVNiXEhIpVXquWsfoysNVlBmyA4V VICVGii1G2WK9kfP7Q82 =UyRB -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Increasing maxThreads results in more "Connection refused" errors?
Chris > -Original Message- > From: Christopher Schultz [mailto:ch...@christopherschultz.net] > Sent: Friday, February 03, 2017 11:08 AM > To: Tomcat Users List > Subject: Re: Increasing maxThreads results in more "Connection refused" > errors? > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > John, > > On 2/3/17 11:17 AM, john.e.gr...@wellsfargo.com.INVALID wrote: > > I'm doing some scalability testing of an app in Tomcat 7.0.73. > > > > Some relevant connector config details: maxThreads=80 > > maxKeepAliveRequests=100 keepAliveTimeout=1 maxConnections > > unspecified (defaults to maxThreads according to the docs) acceptCount > > unspecified (100 according to the docs) clientAuth=true > > Which connector? > > > FWIW, I'm testing two Tomcat instances on the same server. They are > > behind a load balancer. > > Which load-balancer using which protocol? > > > It appears that when the load generator tries to exceed maxThreads, > > the new connection rate goes up quickly and CPU usage shoots up with > > it. > > When you say the "new connection rate" goes up, what exactly do you mean? > When using a load-generator, I would certainly expect the "new connection > rate" to go up. > > > I assume this is because Tomcat is proactively closing idle keep alive > > connections to service new connections. > > No. Tomcat behaves the same under load as when not under load. If you are > using the BIO connector (it sounds like you are), then each connection will > have > to wait for the KeepAlive timeout to expire before servicing another request > from another client. > > This is complicated by the reverse-proxy and so those details are important to > provide. > > > In an effort to keep the CPU in check, I tried increasing maxThreads > > from 80 to 120. > > If the CPU is very busy, allowing more threads to run is counter-productive, > isn't > it? > > > This seemed to work well in a lot of ways. New connection rate didn't > > increase as much, CPU didn't increase as much, there was more > > connection reuse (more requests per connection,) and response times > > didn't deteriorate as much. > > > > Great, right? > > > > Then I noticed a large increase in "Connection refused" errors on the > > load generator. In other words, a higher maxThreads also results in a > > high error rate. The total hits per second from the client's > > perspective is about 60 in both cases. With maxThreads=80, there are > > about 3 connection refused errors per second at that volume. With > > maxThreads=120, there are about 10 connection refused errors per > > second. > > Are you hitting the load-balancer or Tomcat directly during your load-test? > > If you are hitting the load-balancer, then the details of that configuration > are > more important for the client: if the lb will only accept 10 connections, it > doesn't > matter what Tomcat is willing to accep t. > > > I have no idea why this is. Can someone explain this or what I can do > > about it? > > Post more data. We'll get to the bottom of it. > > - -chris Thanks for asking those good questions. When I checked with the load balancer people, they told me there was a 70 connection limit to each Tomcat server enforced by the load balancer. It took a while to get that limit removed, but now that it's gone, the connection errors completely went away. I think the limit is a good idea and we will probably re-establish a higher one after additional testing. Thanks John