Hello, The issue: TomEE is generating HTTP/1.1 500 errors after a period of time, which is approximately once 100+ concurrent users are processing data.
All servers are Windows 2012 R2 Standard Front-end=Apache 2.4.12\mod_jk 1.2.40 for load-balancing Back-end=TomEE 1.7.0 instances for workflow processing using JDK 8.0_91 This setup is for internal end-users only, no access from the outside. Also, this setup is not for displaying web pages. The end-users access data in a Sybase database via an Interactive client-side application. The TomEE instance servers process that data workflow that is then saved back to the database. There are two back-end servers utilizing two TomEE instances per server. This was changed to three TomEE instances each, but this only prolonged getting the HTTP errors slightly. Both TomEE servers have the same resources; all TomEE instances are configured the same. The problem occurs only when the Apache requests reach more than 100 concurrent users. The Apache winnt mpm was configured for 500 ThreadsPerChild. Load balancing factor=1. The load balancing method was at the default, which is Request. The TomEE instance ajp 1.3 connector attributes only defined the port #, protocol and redirect port #. Other attributes would have been at default. Setting log options to debug causes the end-users’ data processing to hang before the 100 concurrent user count is reached. >From the testing that I am able to do with JMeter, it appears that the TomEE ajp connector default thread limit has been reached, not allowing new sockets to be opened. No errors are posted in the catalina logs when this happens. The Apache and TomEE access logs only show the errors as HTTP/1.1 500 errors. The Apache errors are posted 30 seconds after the TomEE errors. This behavior can be seen using the JK Status Manager. This is a 10 minute test run using 225 simulated users requesting the broker wsdl for each instance. The first error under the Err column was generated at a little over the 1 minute mark so it would have been an HTTP 500 error for an end-user. This is the same, test after test after test although the problem can show up in any instance, not just the “A” one shown here. <http://tomee-openejb.979440.n4.nabble.com/file/n4680255/Image1.png> Adding these settings to TomEE’s ajp connectors, acceptCount="150" maxThreads="300" minSpareThreads="50" And upping the JMeter users to 250 over 10 minutes produced these results. Still getting strange, uneven behavior on the connectors but no errors in the Err column. <http://tomee-openejb.979440.n4.nabble.com/file/n4680255/Image2.png> Next, changing the load-balancer method from the default Request to Busyness produced these results. Still no errors and far better connector loads. Also, note that the load-balancer Value has decreased dramatically. <http://tomee-openejb.979440.n4.nabble.com/file/n4680255/Image3.png> The positive results from these changes have been consistent over many, many tests. Still, since I can’t reproduce the same load in testing that the end-users do, I cannot put this TomEE setup back into production without a high percentage of assurance that it will not fail again due to the same problem. Can anyone think of or suggest some further configuration changes that I should research or even some other tuning that I could test? Thanks, Jim -- View this message in context: http://tomee-openejb.979440.n4.nabble.com/HTTP-Errors-on-Load-Balanced-TomEE-Servers-tp4680255.html Sent from the TomEE Users mailing list archive at Nabble.com.
