Hello all,
I'm running tomcat 6.0.32 on Cent OS 6 with 2 front end apache load balancers
with a firewall in between the tomcat and load balancers using mod_jk v.
1.2.37 under apache 2.2.10 to connect the backend tomcat. I have had this
running ok for a few years but our user traffic has increased significantly.
A few months ago, the tomcat server seemed to refuse or not accept any new
connections from either load balancer and required a restart on the tomcat end,
even though I could easily connect to tomcat on port 8080(manager). I can
intermittently telnet to port 8009, but am denied a bit as well both inside and
outside the firewall.
I proceeded to split the tomcats up into their own instances, hoping when this
issue recurred that it would only affect a particular tomcat app. It also gave
our developers the ability to patch a single tomcat app without downing all of
our apps.
Unfortunately, this issue has recurred several times and I have spent most of
my days researching and digging for hope of someone with a similar experience
that may have resolved it. Last Friday the problem was so bad, I had to
completely restart the tomcat server(reboot it).
So far I am at a loss...I have installed psi-probe on all tomcat instances to
give me more in depth analysis to tomcat threads and related server metadata
when the problem is occuring. I have made a few modifications to
workers.properties, in particular to decrease the connection timeout as well as
the tomcat ajp connector from 10 minutes to 5 minutes and added the ping
timeout and socket timeout. I also increased my apache prefork MPM client
connections to 500 on each load balancer. Below is my relevant configs...any
suggestions to help remedy this would help... I have also increased threads
from 200 to 500 on all tomcat instances.
Workers.properties:
worker.list=jkstatus,server1,server2,server3,server4,server5,server6,server7,server8
worker.jkstatus.type=status
# Let's define some defaults
worker.basic.port=8009
worker.basic.type=ajp13
worker.basic.socket_keepalive=True
worker.basic.connection_pool_timeout=300
worker.basic.ping_timeout=1000
worker.basic.ping_mode=A
worker.basic.socket_timeout=10
worker.lb1.distance=0
worker.lb1.reference=worker.basic
worker.server1.host= server1hostname
worker.server1.reference=worker.lb1
worker.server2.host=server2hostname
worker.server2.reference=worker.lb1
worker.server3.host=server3hostname
worker.server3.reference=worker.lb1
worker.server4.host= server4hostname
worker.server4.reference=worker.lb1
worker.server5.host= server5hostname
worker.server5.reference=worker.lb1
worker.server6.host= server6hostname
worker.server6.reference=worker.lb1
worker.server7.host= server7hostname
worker.server7.reference=worker.lb1
worker.server8.host= server7hostname
worker.server8.reference=worker.lb1
httpd.conf:
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 500
MaxClients 500
MaxRequestsPerChild 5000
</IfModule>
Tomcat server.xml:
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" address="x.x.x.x" protocol="AJP/1.3"
redirectPort="8443" connectionTimeout="300000" maxThreads="500" />
