Hi John,
John Moore schrieb:
Rainer Jung wrote:
Please have a look at your mod_jk log file and look for '[error]'
messages. Around the error messages there should also be additional
info messages giving some mor einfo about the root cause.
mod_jk.log is empty.
VERY strange. Set JkLogLevel to info. When using 1.2.26: It should at
least log the version number during startup. If it doesn't, then there's
already a logging problem to fix.
OK, you could switch to mod_jk 1.2.26 so we don't have to check
against any old bugs. 1.2.26 is now 3 months old and no major new bugs
are known.
OK, I'll do that this weekend..
That sounds really weird. If you stop and start Apache httpd and
Tomcat (i.e. all processes completely stopped, wait a few seconds and
then started freshly), those processes should not have any more
knowledge of previous problems.
That is what I would have thought.. no joy... I attached the netstat
output so you can see all of the lines for yourself.. Hope that helps..
There is no LISTEN for 8009 in this output. So once you get into this
state we wouldn't expect mod_jk to be able to reach Tomcat with new
connections anymore.
Why do we get into such a situation? Since you are using a 64Bit VM, I
really don't know :(
Which connector do you use? The usual Coyote (Jav) connector, or the
native APR connector (also called tcnative)?
Possibly related information:
netstat -np --tcp reports dozens of "dead" connections to port 8009,
like the following samples: (count in parens to save space here):
tcp 0 0 127.0.0.1:8009
127.0.0.1:48461 SYN_RECV - (13)
tcp 0 6 127.0.0.1:48459
127.0.0.1:8009 FIN_WAIT1 - (3) tcp 6
0 ::ffff:127.0.0.1:8009 ::ffff:127.0.0.1:48191 CLOSE_WAIT
- (50)
The CLOSE_WAIT most likely come from the fact, that you use a
connect_timeout (that's fine) for mod_jk, but have no
connectionTimeout on teh Toncat side. Set connectionTimeout in the
Tomcat sidfe in the AJP connector to 600000 (milliseconds).
Actually I do.. Here is that setting from my server.xml
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009"
enableLookups="false"
redirectPort="8443"
protocol="AJP/1.3"
connectionTimeout="60000"
maxThreads="750"
debug="0" />
Is there anything else here that needs changing..?
Hmmm, maybe set connectionTimeout to 600000 instead of 60000. Otherwise
TC and mod_jk don't use the same timeout (I think you had 10 minutes ofr
mod_jk). This will not be the reason for your strange observation, but
it should reduce the CLOSE_WAIT connections during normal operations.
The SYN_RECV seem to indicate, that your Tomcat is not able to accept
any more connections. Two possible reasons:
- all threads in the thread pool of your AJP connector (Tomcat side)
are already connected with httpd and no more threads are left. E.g. in
your case it could be, that 50 threads are connected and maybe the
size of your thread pool is 50?
- you got an exception during creation of a new thread for the pool,
and as a consequence your Tomcat is not listening any more. Can you
see the LISTEN for 8009 in netstat? If no, the most likely cause is,
that your JVM exceeded the number of threads that it can create (most
32 Bit JVMs: something between 400 and 500).
This is a 64 bit system and using 64 bit JVM... I was seeing this
problem also on a 32 bit system and had hoped that the 64 bit JVM would
give me more "margin of error", apparently not. (See attached netstat
output)
If the problem happens again: check to see, if you can telnet to the
TC port (telnet localhost 8009). If no, then mod_jk also has no chance
of connecting to the Tomcat and we must find out, why Tomcat doesn't
accept any more connections. In that case a thread dump of the Tomcat
java process is helpful (kill -QUIT, result goes to catalin.out).
Good suggestion, I'll give that a try..
Thanks for all of the other suggestions, I made those changes
immediately, and I'll upgrade the connector this weekend. If you are
anyone else have some ideas for me to try, I would REALLY appreciate it..
I would run a cron job, that writes out the netstat statistics for 8009
(how many connections in which state, including LISTEN etc.) once a
minute to get an idea, if you run into something extreme immediately
before the LISTEN dies.
The previous hint about taking thread dumps once the problem is there is
still valid. That way we can check from the inside of the JVM, if the
accept thread is gone. But we still won't know why.
Any interesting log items on the TC side? OutOfMemory errors?
Those port 3050 connections are DB connections or something similar I
suppose?
Thanks again..
John..
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
