We saw a strange production issue this morning that seems to be related
to Apache/mod_jk but I wanted to see if anyone can shed some lght on it.
First, the environment:
2 Redhat 7.3 Linux server talking to 2 Sun Solaris 8 Tomcat 4.0.6
servers. Each server has 2 separate Tomcat instances for a total of 4.
The Java uses Oracles JDBC drivers (not through Tomcat, but through a
custom connection pool) to talk to an Oracle DB runnin on AIX 4.x.
The first symptom was a huge spike in DB usage due to the creation of
around 35 pooled connections to the database, This led us to think
something happened on the database side, but it seems that the pool only
grew on one of the Tomcat instances; the other instances showed the
default pool size. The Oracle database did not show any locked tables or
huge, hanging queries.
The only problems I see in logs are related to mod_jk and Tomcat.
First in the Tomcat engine log,I see:
2002-12-23 09:07:14 Ajp13Connector[12009] No processor available, rejecting this
connection
There are hundreds of these messages. Then I start seeing this message
in the engine log:
2002-12-23 09:07:28 Ajp13Processor[12009][18] process: invoke
java.io.IOException: Broken pipe
at java.net.SocketOutputStream.socketWrite(Native Method)
at java.net.SocketOutputStream.write(SocketOutputStream.java:91)
at org.apache.ajp.Ajp13.send(Ajp13.java:525)
at org.apache.ajp.RequestHandler.finish(RequestHandler.java:501)
at org.apache.ajp.Ajp13.finish(Ajp13.java:395)
at org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:196)
at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:464)
at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551)
at java.lang.Thread.run(Thread.java:479)
About a minute after, in the mod_jk logs, I see:
[Mon Dec 23 09:08:29 2002] [jk_ajp_common.c (679)]: ajp_connection_tcp_get_message:
Error - jk_tcp_socket_recvfull failed
[Mon Dec 23 09:08:29 2002] [jk_ajp_common.c (1041)]: Error reading reply
[Mon Dec 23 09:08:29 2002] [jk_ajp_common.c (1178)]: In jk_endpoint_t::service,
ajp_get_reply failed in send loop 0
This goes on for about 20 seconds before I see a huge number of the
following:
[Mon Dec 23 09:08:49 2002] [jk_ajp_common.c (961)]: Error ajp_process_callback -
write failed
I am at a loss at what might cause this. Could it be related to
soemthing in Tomcat? What exactly does this sequence of events tell me?
Any light one can shed would be greatly appreciated.
Ben Ricker
--
Ben Ricker <[EMAIL PROTECTED]>
Wellinx.com
--
To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>
