Hi Experts,
I have the following setup:
Apache1
/ \
Apache2 Apache3
\ /
Tomcat1
All Apaches are version 2.2.3 (RedHat)
Apache2 and Apache3 loadbalance Tomcat1 (6.0.18) with mod_jk (2.2.28).
In idle state the AJP connector of Tomcat1 only has 7 active connections.
After launching a stress test of Tomcat1, it's AJP connector has reached
maxThreads (200). After the stress test has finished, there are still 200
active connections in the AJP connector.
Because of this, apache2 and apache2 cannot receive any heartbeat message
anymore from the AJP connector and mark Tomcat1 as dead. I can access
perfectly through port 8080, so Tomcat1 isn't dead at all!
According to a threaddump, ALL TP-Processor-x threads are in the following
state:
"TP-Processor41" daemon prio=10 tid=0x8ada7800 nid=0x1496 runnable
[0x861eb000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0xbab3ef10> (a java.io.BufferedInputStream)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:620)
at
org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:558)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:685)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:619)
LambaProbe confirms this and shows 200 AJP connections in KeepAlive state.
Please see my mod_jk and tomcat ajp connector configurations:
server.xml:
<Connector port="9009" protocol="AJP/1.3" redirectPort="8443"
URIEncoding="UTF-8" />
workers.properties:
worker.list=liferay_alf
worker.alfresco1.port=9009
worker.alfresco1.host=10.133.23.4
worker.alfresco1.type=ajp13
worker.alfresco1.lbfactor=1
worker.liferay_alf.type=lb
worker.liferay_alf.balance_workers=alfresco1
worker.liferay_alf.sticky_session=0
In the Tomcat documentation I've seen that the AJP connector has a property
keepAliveTimeout. Although, before setting this, I would like to know from
you guys whether this looks as apache2 and apache3 continuously keep sending
keepAlive messages to the open AJP connections. If so, why is this? And do
you think setting keepAliveTimeout in Tomcat is the best solution?
Thanks!
