mod_jk hangs

[Shawn Wilson]

Just yesterday we started running into problems requesting pages on our production website through mod_jk. Previously everything had been working fine for months. Essentially what happens is when you request the page for the first time (doesn't seem to matter what browser you're using) the browser just sits and spins. Then after hitting refresh (Ctrl+R) a couple times it finally comes up. Subsequent visits to the same page continue to work for me in Mozilla but after closing IE and reopening it after a couple more requests it hangs again. In case it helps, the site in question is http://www.atmreports.com.

I simply cannot figure out what is going on. I do see many errors in the mod_jk.log, but these had been there before when everything worked fine too:

[Wed May 05 09:27:46 2004] [jk_ajp_common.c (1137)]: Error reading reply from tomcat. Tomcat is down or network problems.
[Wed May 05 09:27:46 2004] [jk_ajp_common.c (1290)]: ERROR: Receiving from tomcat failed, recoverable operation. err=0
[Wed May 05 09:27:46 2004] [jk_ajp_common.c (1309)]: sending request to tomcat failed in send loop. err=0
[Wed May 05 09:27:46 2004] [jk_ajp_common.c (738)]: ERROR: can't receive the response message from tomcat, network problems or tomcat is down. err=-1

A few things to note, any of which may contribute to the problem but for the life of me I can't figure out how:

1. On Friday we just moved out datacenter to a new location. However we see this problem even internally at the location so I don't believe it is any sort of routing issue (and the problem didn't start until Tuesday).

2. The clock on this webserver was many hours off. On Tuesday (yes, the day the problems started) the clock was synchronized and the timezone correctly set so the machine now has correct time. This seems to be the most probable cause simply because it happened around the same time, but I just can't understand why or how.

3. Yesterday we rebooted the box to see if that would help. It did. We didn't have any more problems at all that day until this morning we see the problems again. Today we recycled just tomcat and that seems to have fixed it as well, though I expect that by tomorrow I will see the problem again.

4. I wouldn't think this could have anything to do with that new Sasser worm, but listing that just in case.

The machine is running Linux kernel 2.4.20-8 with httpd-2.0.40-21 and jakarta-tomcat-5.0.18. We are running 4 instances of tomcat on the box, each started with the following JVM options:

-Xmn128M -Xmx1024M -Djava.awt.headless=true

Nothing having to do with configuration has changed at all in a long time. Here is the relevant configuration sections:

# Mod_jk (Tomcat connector)
LoadModule jk_module modules/mod_jk.so

<IfModule mod_jk.c>
    JkWorkersFile /etc/httpd/conf/workers.properties
    JkLogFile /var/log/httpd/mod_jk.log
    JkLogLevel info
    JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
    JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
    JkRequestLogFormat "%w %V %T"
</IfModule>

...and later (in a <VirtualHost> section)...

    JKMount /myreports balancer
    JKMount /myreports/* balancer

Here is our workers.properties file (comments removed):

workers.java_home=/usr/java/j2sdk1.4.2_03
ps=/
worker.list=balancer

worker.ajp13_0.port=8090
worker.ajp13_0.host=localhost
worker.ajp13_0.type=ajp13
worker.ajp13_0.lbfactor=1

worker.ajp13_1.port=8091
worker.ajp13_1.host=localhost
worker.ajp13_1.type=ajp13
worker.ajp13_1.lbfactor=1

worker.ajp13_2.port=8092
worker.ajp13_2.host=localhost
worker.ajp13_2.type=ajp13
worker.ajp13_2.lbfactor=1

worker.ajp13_3.port=8093
worker.ajp13_3.host=localhost
worker.ajp13_3.type=ajp13
worker.ajp13_3.lbfactor=1

worker.balancer.type=lb
worker.balancer.balanced_workers=ajp13_0, ajp13_1, ajp13_2, ajp13_3
worker.balancer.sticky_session=1

Any help would be greatly appreciated!

Thanks,
-shawn

--
====================================
Shawn Wilson [EMAIL PROTECTED]
Software Developer, ATMReports.com
PH: 877-327-0873, FAX: 406-294-5806
====================================

smime.p7s

Description: S/MIME Cryptographic Signature