Hi,
this seems also to be a problem in mod_jk , tomcat4.1.10, apache 1.3 on a linux server however it does build up slowly over the day.
The problem seems to be that the connection via mod_jk (Port 8009) does not close, so the java/tomcat processes
will not quit after responding to the request (or the other way around?). This problem occurs only in
heavy load situations. After that I see many open socket connections an some tomcat processes that will not quit.
Snapshot of the logfiles:
tomcat / catalina.out: Failed to authentice as null Error number: 50
tomcat / catalna_log:
2004-10-22 08:52:29 Ajp13Processor[8009][358] Starting background thread
2004-10-22 08:52:34 Ajp13Processor[8009][359] Starting background thread
2004-10-22 08:52:34 Ajp13Connector[8009] No processor available, rejecting this connection
2004-10-22 08:52:34 Ajp13Connector[8009] No processor available, rejecting this connection
apache / mod_jk
[Fri Oct 22 08:52:34 2004] [jk_ajp_common.c (738)]: ERROR: can't receive the response message from tomcat, network problems or tomcat is down. err=-104
[Fri Oct 22 08:52:34 2004] [jk_ajp_common.c (1137)]: Error reading reply from tomcat. Tomcat is down or network problems.
Is this a mod_jk problem not closing the connections or a tomcat problem not telling mod_jk that it's finished or a system problem not
releasing the sockets ?
Thanks for any ideas... Chris
David Smith wrote:
Hmmm.. My assumption to your email was their was some kind of possible probing of your Apache server and/or a misbehaving client. Do you run snort on the box hosting the Apache httpd service? It's an intrusion detection tool designed to log suspicious activity. That could be something to look at. Otherwise I can't think of a reason in the world for httpd to spontaneously go full out. It's not something I've ever seen.
--David
Lars George wrote:
David,
This proves more difficult, since the requests look like standard requests that work at other times. Moreover the POST data is no logged anyway so I cannot check if it was a value that was sent in by chance.
Is there anything else I can check to see what is going on? I was more thinking along the lines of using some low-level tools to see if everything is OK, for example critical resources related to Apache or Tomcat (note, they are on separate machines). For example I tried using "netstat -p" on either side to see what the connections are saying. I als did a thread dump of the Tomcat JVM to see what the Coyote connectors and all the other threads are up to. But There is nothing conclusive so far. Just out of the blue the apache runs full, and there seems no relation to when during the day or how high the load etc.
Thanks, Lars
David Smith wrote:
I would start with the apache logs and find out what kind of requests are logged in the access just before the event. That should get you going in the right direction.
--David
Lars George wrote:
Hi,
We have an odd problem we cannot solve. Maybe someone else has come across this too.
We use Apache 2.0.52 with the Tomcat 5.0.25 and its included mod_jk2 with Coyote/JK2 AJP 1.3 connector.
Usually Apache uses 230 slots out of 1000 it has set as the maximum. This can be seen from the server-status page, it equals to say 32 requests per second.
Then all of sudden the Apache runs full and exceeds all empty slots, ie. goes up to 1000 requests currently being processed and accepts no further connections. All but a handful of these slots are in state "W". Looking at the URI's the requests are both static content (like images) served directly by Apache, but of course many requests are to the dynamic content created by the app server (Tomcat).
I do not think this is a load issue per se, ie. where we have someone hammering us with requests, since looking at the connections and URL's they are all so random but valid at the same time that I do not think this is it.
What puzzles me more is that they are all in the state "W". which means "sending reply". The network seems to be OK at that point of time too since I can connect to the machines no problem.
Only if I restart Apache by ultimately killing (killall -9 httpd) all processes I can revive the site. After the Apache restart everything goes back to normal right away. This seems to mean something too, at least that it seems to be getting stuck somehow but whatever caused it does not seem to exist anymore. Like something short, for example a specific request, caused this.
What could I check from here? I cannot take the Apache out since we need it for rewriting and session handling.
Any help or pointers are greatly appreciated.
Thanks, Lars
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
