Hi,
I'm trying to track down some errors with our apache tomcat
system and not having a great deal of luck.
We had an issue where the mod_jk connectors would loose track
of the tomcat connectors and just start refusing requests with error 500. This
was with an apache front end and tomcat backend.
We seem to have the majority of this sorted, so now I am trying
to clean up the last few problems. Namely all these errors:
[Fri Aug 22 14:38:33.369 2008] [15192:2534538208] [info]
ajp_send_request::jk_ajp_common.c (1347): (TLC-PROD) all endpoints are
disconnected, detected by connect check (1), cping (0), send (0)
[Fri Aug 22 14:38:33.901 2008] [19082:2534538208] [info]
ajp_send_request::jk_ajp_common.c (1347): (LOGIN-PROD) all endpoints are
disconnected, detected by connect check (1), cping (0), send (0)
[Fri Aug 22 14:38:33.941 2008] [15184:2534538208] [info]
ajp_send_request::jk_ajp_common.c (1347): (LOGIN-PROD) all endpoints are
disconnected, detected by connect check (1), cping (0), send (0)
[Fri Aug 22 14:38:36.159 2008] [15243:2534538208] [info]
ajp_send_request::jk_ajp_common.c (1347): (TLC-PROD) all endpoints are
disconnected, detected by connect check (1), cping (0), send (0)
[Fri Aug 22 14:38:36.394 2008] [22157:2534538208] [info]
ajp_send_request::jk_ajp_common.c (1347): (TLC-PROD) all endpoints are
disconnected, detected by connect check (1), cping (0), send (0)
These seem to be relatively benign, but they annoy me and as
far as I tell should not occur.
After a bit of reformatting of the mod_jk log I ended up with
these snippet as an example:
[9641:2534538208] (MY-PROD) [Fri Aug 22 12:11:35.193 2008]
[9641:2534538208] (MY-PROD) [Fri Aug 22 12:15:35.138 2008]
[9641:2534538208] (MY-PROD) [Fri Aug 22 12:17:09.159 2008]
[9641:2534538208] (MY-PROD) [Fri Aug 22 12:17:25.898 2008]
[9641:2534538208] (MY-PROD) [Fri Aug 22 12:18:18.216 2008]
[9641:2534538208] (MY-PROD) [Fri Aug 22 12:18:48.942 2008]
This seems to indicate that a single process 9461 is finding
its endpoint to the MY-PROD worker disappearing on a regular basis. There are
even drops ~20 secs apart.
As far as I can tell there should be 1 worker per apache process (I'm using
prefork). We run 12 different workers for this single apache instance. Does
this mean 1 of each of the 12 workers can serve each Apache process at a time
or is it 1 worker at a time total per apache process?
I could understand the number of disconnects of the apache processes have to
keep swapping between workers, but it has me worried if there is no need to do
this.
The configuration is setup such that apache should hold a pool open for 10
minutes at idle. Apache processes are not required to close they have a 0 max
requests set. I have timeouts configured correctly I hope for the connectors,
here are the relevant snippets:
# global configurations
worker.template.type=ajp13
worker.template.lbfactor=1
worker.template.connection_pool_timeout=600
worker.template.connect_timeout=5000
worker.template.prepost_timeout=5000
worker.MY-PROD.port=8010
worker.MY-PROD.host=127.0.0.1
worker.MY-PROD.reference=worker.template
<Connector port="8010"
maxThreads="300"
minSpareThreads="80"
maxSpareThreads="50"
connectionTimeout="600000"
enableLookups="false"
redirectPort="-1"
protocol="AJP/1.3" />
I have also set JKShmSize to a much larger number so that there is no
possibility of running out of space. How many workers actually use this? I
have 12 configured in my apache instance. Is this 12 total? Or is it 1 worker
per process, so a max of 254? Or is it the worst case of 12 worker definitions
* 254 possible processes?
If there is also some sort of nuts and bolts how tomcat works at a lower level
type document? Part of my problem may stem from the way I'm trying to guess
underlying behaviour from config options.
Any help much appreciated,
Dave
