Once an Apache process opens a socket connection to Tomcat for Ajp that
connection remains open until that httpd process dies. That way the
httpd process can reuse that connection. For Apache 1.3 there is a
one to one mapping between apache httpd processes and Tomcat Ajp13Processors.
You can end up with as many Ajp13Processors as you have httpd processes.
So the MaxProcessors config needs to be greater than the max number
of httpd processes used by apache. And each Ajp13Processor runs in
its own thread waiting for a request from Apache.
Regards,
Glenn
Lindsay Patten wrote:
Thanks for the pointer Glenn, it was much appreciated!
So, I've been doing thread stacktrace dumps, but what I found doesn't
make a lot of sense to me.
After restarting tomcat there were 5 processor threads all blocked on a
monitor:
"Ajp13Processor[8085][0]" daemon prio=1 tid=0x0x4f801b28 nid=0x45ae
waiting on m
onitor [bd9ff000..bd9ff8ac]
at java.lang.Object.wait(Native Method)
- waiting on <0x446602c0> (a org.apache.ajp.tomcat4.Ajp13Processor)
at java.lang.Object.wait(Object.java:426)
at
org.apache.ajp.tomcat4.Ajp13Processor.await(Ajp13Processor.java:305)
- locked <0x446602c0> (a org.apache.ajp.tomcat4.Ajp13Processor)
at
org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:490)
at java.lang.Thread.run(Thread.java:536)
I presume these are waiting for the scheduler thread to pass them a
request.
A while later a bunch of background threads get spawned and I do another
dump/trace,
now there are a whole bunch of threads in a runnable state trying to
read request packets:
"Ajp13Processor[8085][21]" daemon prio=1 tid=0x0x8289b38 nid=0x70d6
runnable [ba
fff000..bafff8ac]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:116)
at org.apache.ajp.Ajp13.readN(Ajp13.java:429)
at org.apache.ajp.Ajp13.receive(Ajp13.java:469)
at org.apache.ajp.Ajp13.receiveNextRequest(Ajp13.java:274)
at
org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:371
)
at
org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:495)
at java.lang.Thread.run(Thread.java:536)
Despite the name receiveNextRequest, it looks like that method is being
called to receive each request including the first. So, the ajp13
processor is sitting blocked waiting for a full ajp13 packet from apache
and apache is not sending data and not closing the connection/socket.
From the comments in the code it sounds like apache restarts
periodically, closing the sockets and unblocking the read calls, at
which point these processors return to the pool - so if apache restarts
before you reach your threads limit you are ok.
What I don't understand is how apache can be starting up ajp13
connections to tomcat and then not sending a full packet without this
causing user visible failures. Shouldn't there be users getting page
loads timing out on them? That doesn't seem to be happening.
Does this indicate a fault in apache or is it normal for apache to send
partial requests without closing the connection?
Any further guidance?
Thanks,
Lindsay
Glenn Nielsen wrote:
One way to start debugging this type of problem is to tell the java
process running Tomcat to do a Thread Stacktrace Dump.
kill -QUIT <java processid>
Then analyze the stack traces for all threads.
Regards,
Glenn
Lindsay Patten wrote:
Hi,
I thought I would take a different tack on my problem with the
Ajp13Processor using up all of its threads problem.
Does anyone know if the Ajp13Processor has a timeout when it is
looking for a worker to handle a request? If the worker threads were
all swapped out and taking a long time to get going and respond,
would the main thread start a new background thread instead of
waiting? Or does it just have a list of available workers and blocks
until the worker it selects responds?
If it is the former that would explain my problem since I have
experienced my processes getting swapped out and taking a long time
to respond. The question would then become: is a way to specify how
long the scheduler should wait before starting a new thread? At least
I would have something I could take to my service provider. On the
other hand if I knew for sure that the scheduler thread didn't have a
timeout on worker threads than I could concentrate on looking
elsewhere. Does anyone know? Or know where to look or ask short of
reading the source?
Thanks,
Lindsay
Lindsay Patten wrote:
Hi,
I am having a problem where tomcat keeps intermitantly starting new
ajp13 processor threads, eventually it reaches the max and starts
refusing connections. My hosting provider (linux machine) provides a
private version of tomcat for each user with a single apache web
server, there are typically a large number of processes on the
machine (>4000). My particular tomcat server (v4.0.3) is not being
loaded at all but the ajp13processor threads don't seem to not get
reused under some circumstances that I don't understand. I have
added logging statements in my jsp pages and the pages appear to run
to completion. Each jsp page accesses some info from a mysql db and
displays it (*Summary) or takes some data from a form and sticks it
in the db (Page). The system will often run for several hours and
hundreds of requests without starting any threads, but then
sometimes, as below, it starts new threads for almost every request
for a while. Is there a way to determine if the threads are getting
hung up or returned to the pool? If a thread were swapped out and
taking a long time to get swapped back in would tomcat start a new
thread? The ps command indicates that the processes are there and
sleeping. Any help or debugging pointers would be greatly
appreciated. Thanks - Lindsay
2002-10-18 03:50:45 Ajp13Processor[8085][14] Starting background
thread
2002-10-18 03:50:53 Ajp13Processor[8085][15] Starting background
thread
2002-10-18 03:50:59 NationalSummary start
2002-10-18 03:51:00 NationalSummary end
2002-10-18 03:51:10 DistrictSummary start
2002-10-18 03:51:10 DistrictSummary end
2002-10-18 03:51:14 Ajp13Processor[8085][16] Starting background
thread
2002-10-18 03:51:14 SubdistrictSummary start 42
2002-10-18 03:51:14 SubdistrictSummary end 42
2002-10-18 03:51:23 Ajp13Processor[8085][17] Starting background
thread
2002-10-18 03:51:23 PageFrame start
2002-10-18 03:51:23 PageFrame end
2002-10-18 03:51:24 Ajp13Processor[8085][18] Starting background
thread
2002-10-18 03:51:24 Page start 206
2002-10-18 03:51:24 Page end 206
2002-10-18 03:53:32 Ajp13Processor[8085][19] Starting background
thread
2002-10-18 03:53:32 Page start 206
2002-10-18 03:53:33 Page end 206
2002-10-18 03:53:56 Ajp13Processor[8085][20] Starting background
thread
2002-10-18 03:53:56 NationalSummary start
2002-10-18 03:53:56 NationalSummary end
2002-10-18 03:54:42 Page start 206
2002-10-18 03:54:42 Page end 206
2002-10-18 03:55:06 DistrictSummary start
2002-10-18 03:55:06 DistrictSummary end
2002-10-18 03:55:24 Ajp13Processor[8085][21] Starting background
thread
2002-10-18 03:55:24 NationalSummary start
2002-10-18 03:55:24 NationalSummary end
2002-10-18 03:55:34 DistrictSummary start
2002-10-18 03:55:34 DistrictSummary end
2002-10-18 03:55:41 Page start 206
2002-10-18 03:55:41 Page end 206
2002-10-18 03:56:52 Ajp13Processor[8085][22] Starting background
thread
2002-10-18 03:56:52 Page start 206
2002-10-18 03:56:52 Page end 206
2002-10-18 03:58:16 Page start 206
2002-10-18 03:58:16 Page end 206
2002-10-18 03:59:31 Page start 206
2002-10-18 03:59:31 Page end 206
2002-10-18 04:00:24 Page start 206
2002-10-18 04:00:24 Page end 206
2002-10-18 04:01:34 Page start 206
2002-10-18 04:01:34 Page end 206
--
To unsubscribe, e-mail: <mailto:tomcat-user-unsubscribe@;jakarta.apache.org>
For additional commands, e-mail: <mailto:tomcat-user-help@;jakarta.apache.org>