Hi it would be interesting to tune the timeouts on the connector(s) - potentially on the LB too.
If it happens again don't forget to take a thread dump of processes, it often helps to see what the server is doing and if it is just busy. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://blog-rmannibucau.rhcloud.com> | Old Wordpress Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber <http://www.tomitribe.com> | JavaEE Factory <https://javaeefactory-rmannibucau.rhcloud.com> 2016-09-29 21:25 GMT+02:00 Jlemoine <[email protected]>: > Hello, > > The issue: TomEE is generating HTTP/1.1 500 errors after a period of time, > which is approximately once 100+ concurrent users are processing data. > > All servers are Windows 2012 R2 Standard > Front-end=Apache 2.4.12\mod_jk 1.2.40 for load-balancing > Back-end=TomEE 1.7.0 instances for workflow processing using JDK 8.0_91 > > This setup is for internal end-users only, no access from the outside. > Also, this setup is not for displaying web pages. The end-users access data > in a Sybase database via an Interactive client-side application. The TomEE > instance servers process that data workflow that is then saved back to the > database. > > There are two back-end servers utilizing two TomEE instances per server. > This was changed to three TomEE instances each, but this only prolonged > getting the HTTP errors slightly. Both TomEE servers have the same > resources; all TomEE instances are configured the same. The problem occurs > only when the Apache requests reach more than 100 concurrent users. The > Apache winnt mpm was configured for 500 ThreadsPerChild. Load balancing > factor=1. The load balancing > method was at the default, which is Request. The TomEE instance ajp 1.3 > connector attributes only defined the port #, protocol and redirect port #. > Other attributes would have been at default. > Setting log options to debug causes the end-users’ data processing to hang > before the 100 concurrent user count is reached. > From the testing that I am able to do with JMeter, it appears that the > TomEE > ajp connector default thread limit has been reached, not allowing new > sockets to be opened. No errors are posted in the catalina logs when this > happens. The Apache and TomEE access logs only show the errors as HTTP/1.1 > 500 errors. The Apache errors are posted 30 seconds after the TomEE > errors. > This behavior can be seen using the JK Status Manager. > This is a 10 minute test run using 225 simulated users requesting the > broker > wsdl for each instance. The first error under the Err column was generated > at a little over the 1 minute mark so it would have been an HTTP 500 error > for an end-user. This is the same, test after test after test although the > problem can show up in any instance, not just the “A” one shown here. > <http://tomee-openejb.979440.n4.nabble.com/file/n4680255/Image1.png> > > > > > > > Adding these settings to TomEE’s ajp connectors, > acceptCount="150" > maxThreads="300" > minSpareThreads="50" > And upping the JMeter users to 250 over 10 minutes produced these results. > Still getting strange, uneven behavior on the connectors but no errors in > the Err column. > <http://tomee-openejb.979440.n4.nabble.com/file/n4680255/Image2.png> > > > > > > > > Next, changing the load-balancer method from the default Request to > Busyness > produced these results. Still no errors and far better connector loads. > Also, note that the load-balancer Value has decreased dramatically. > <http://tomee-openejb.979440.n4.nabble.com/file/n4680255/Image3.png> > > > > > > > > The positive results from these changes have been consistent over many, > many > tests. > Still, since I can’t reproduce the same load in testing that the end-users > do, I cannot put this TomEE setup back into production without a high > percentage of assurance that it will not fail again due to the same > problem. > Can anyone think of or suggest some further configuration changes that I > should research or even some other tuning that I could test? > > Thanks, > Jim > > > > > -- > View this message in context: http://tomee-openejb.979440. > n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-tp4680255.html > Sent from the TomEE Users mailing list archive at Nabble.com. >
