Re: Ajp13Processor starting background threads under low load

[Lindsay Patten]

Hi,
I thought I would take a different tack on my problem with the 
Ajp13Processor using up all of its threads problem.
Does anyone know if the Ajp13Processor has a timeout when it is looking 
for a worker to handle a request? If the worker threads were all swapped 
out and taking a long time to get going and respond, would the main 
thread start a new background thread instead of waiting? Or does it just 
have a list of available workers and blocks until the worker it selects 
responds?
If it is the former that would explain my problem since I have 
experienced my processes getting swapped out and taking a long time to 
respond. The question would then become: is a way to specify how long 
the scheduler should wait before starting a new thread? At least I would 
have something I could take to my service provider. On the other hand if 
I knew for sure that the scheduler thread didn't have a timeout on 
worker threads than I could concentrate on looking elsewhere.  Does 
anyone know? Or know where to look or ask short of reading the source?

Thanks,
   Lindsay

Lindsay Patten wrote:

Hi,
I am having a problem where tomcat keeps intermitantly starting new 
ajp13 processor threads, eventually it reaches the max and starts 
refusing connections. My hosting provider (linux machine) provides a 
private version of tomcat for each user with a single apache web 
server, there are typically a large number of processes on the machine 
(>4000). My particular tomcat server (v4.0.3) is not being loaded at 
all but the ajp13processor threads don't seem to not get reused under 
some circumstances that I don't understand. I have added logging 
statements in my jsp pages and the pages appear to run to completion. 
Each jsp page accesses some info from a mysql db and displays it 
(*Summary) or takes some data from a form and sticks it in the db 
(Page). The system will often run for several hours and hundreds of 
requests without starting any threads, but then sometimes, as below, 
it starts new threads for almost every request for a while. Is there a 
way to determine if the threads are getting hung up or returned to the 
pool? If a thread were swapped out and taking a long time to get 
swapped back in would tomcat start a new thread? The ps command 
indicates that the processes are there and sleeping. Any help or 
debugging pointers would be greatly appreciated.  Thanks - Lindsay

   2002-10-18 03:50:45 Ajp13Processor[8085][14] Starting background 
thread
   2002-10-18 03:50:53 Ajp13Processor[8085][15] Starting background 
thread
   2002-10-18 03:50:59 NationalSummary start
   2002-10-18 03:51:00 NationalSummary end
   2002-10-18 03:51:10 DistrictSummary start
   2002-10-18 03:51:10 DistrictSummary end
   2002-10-18 03:51:14 Ajp13Processor[8085][16] Starting background 
thread
   2002-10-18 03:51:14 SubdistrictSummary start 42
   2002-10-18 03:51:14 SubdistrictSummary end 42
   2002-10-18 03:51:23 Ajp13Processor[8085][17] Starting background 
thread
   2002-10-18 03:51:23 PageFrame start
   2002-10-18 03:51:23 PageFrame end
   2002-10-18 03:51:24 Ajp13Processor[8085][18] Starting background 
thread
   2002-10-18 03:51:24 Page start 206
   2002-10-18 03:51:24 Page end 206
   2002-10-18 03:53:32 Ajp13Processor[8085][19] Starting background 
thread
   2002-10-18 03:53:32 Page start 206
   2002-10-18 03:53:33 Page end 206
   2002-10-18 03:53:56 Ajp13Processor[8085][20] Starting background 
thread
   2002-10-18 03:53:56 NationalSummary start
   2002-10-18 03:53:56 NationalSummary end
   2002-10-18 03:54:42 Page start 206
   2002-10-18 03:54:42 Page end 206
   2002-10-18 03:55:06 DistrictSummary start
   2002-10-18 03:55:06 DistrictSummary end
   2002-10-18 03:55:24 Ajp13Processor[8085][21] Starting background 
thread
   2002-10-18 03:55:24 NationalSummary start
   2002-10-18 03:55:24 NationalSummary end
   2002-10-18 03:55:34 DistrictSummary start
   2002-10-18 03:55:34 DistrictSummary end
   2002-10-18 03:55:41 Page start 206
   2002-10-18 03:55:41 Page end 206
   2002-10-18 03:56:52 Ajp13Processor[8085][22] Starting background 
thread
   2002-10-18 03:56:52 Page start 206
   2002-10-18 03:56:52 Page end 206
   2002-10-18 03:58:16 Page start 206
   2002-10-18 03:58:16 Page end 206
   2002-10-18 03:59:31 Page start 206
   2002-10-18 03:59:31 Page end 206
   2002-10-18 04:00:24 Page start 206
   2002-10-18 04:00:24 Page end 206
   2002-10-18 04:01:34 Page start 206
   2002-10-18 04:01:34 Page end 206




--
To unsubscribe, e-mail:   

For additional commands, e-mail: 





--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: Ajp13Processor starting background threads under low load

[Glenn Nielsen]

One way to start debugging this type of problem is to tell the java
process running Tomcat to do a Thread Stacktrace Dump.

kill -QUIT 

Then analyze the stack traces for all threads.

Regards,

Glenn


Lindsay Patten wrote:

Hi,
I thought I would take a different tack on my problem with the 
Ajp13Processor using up all of its threads problem.
Does anyone know if the Ajp13Processor has a timeout when it is looking 
for a worker to handle a request? If the worker threads were all swapped 
out and taking a long time to get going and respond, would the main 
thread start a new background thread instead of waiting? Or does it just 
have a list of available workers and blocks until the worker it selects 
responds?
If it is the former that would explain my problem since I have 
experienced my processes getting swapped out and taking a long time to 
respond. The question would then become: is a way to specify how long 
the scheduler should wait before starting a new thread? At least I would 
have something I could take to my service provider. On the other hand if 
I knew for sure that the scheduler thread didn't have a timeout on 
worker threads than I could concentrate on looking elsewhere.  Does 
anyone know? Or know where to look or ask short of reading the source?

Thanks,
   Lindsay

Lindsay Patten wrote:

Hi,
I am having a problem where tomcat keeps intermitantly starting new 
ajp13 processor threads, eventually it reaches the max and starts 
refusing connections. My hosting provider (linux machine) provides a 
private version of tomcat for each user with a single apache web 
server, there are typically a large number of processes on the machine 
(>4000). My particular tomcat server (v4.0.3) is not being loaded at 
all but the ajp13processor threads don't seem to not get reused under 
some circumstances that I don't understand. I have added logging 
statements in my jsp pages and the pages appear to run to completion. 
Each jsp page accesses some info from a mysql db and displays it 
(*Summary) or takes some data from a form and sticks it in the db 
(Page). The system will often run for several hours and hundreds of 
requests without starting any threads, but then sometimes, as below, 
it starts new threads for almost every request for a while. Is there a 
way to determine if the threads are getting hung up or returned to the 
pool? If a thread were swapped out and taking a long time to get 
swapped back in would tomcat start a new thread? The ps command 
indicates that the processes are there and sleeping. Any help or 
debugging pointers would be greatly appreciated.  Thanks - Lindsay

   2002-10-18 03:50:45 Ajp13Processor[8085][14] Starting background 
thread
   2002-10-18 03:50:53 Ajp13Processor[8085][15] Starting background 
thread
   2002-10-18 03:50:59 NationalSummary start
   2002-10-18 03:51:00 NationalSummary end
   2002-10-18 03:51:10 DistrictSummary start
   2002-10-18 03:51:10 DistrictSummary end
   2002-10-18 03:51:14 Ajp13Processor[8085][16] Starting background 
thread
   2002-10-18 03:51:14 SubdistrictSummary start 42
   2002-10-18 03:51:14 SubdistrictSummary end 42
   2002-10-18 03:51:23 Ajp13Processor[8085][17] Starting background 
thread
   2002-10-18 03:51:23 PageFrame start
   2002-10-18 03:51:23 PageFrame end
   2002-10-18 03:51:24 Ajp13Processor[8085][18] Starting background 
thread
   2002-10-18 03:51:24 Page start 206
   2002-10-18 03:51:24 Page end 206
   2002-10-18 03:53:32 Ajp13Processor[8085][19] Starting background 
thread
   2002-10-18 03:53:32 Page start 206
   2002-10-18 03:53:33 Page end 206
   2002-10-18 03:53:56 Ajp13Processor[8085][20] Starting background 
thread
   2002-10-18 03:53:56 NationalSummary start
   2002-10-18 03:53:56 NationalSummary end
   2002-10-18 03:54:42 Page start 206
   2002-10-18 03:54:42 Page end 206
   2002-10-18 03:55:06 DistrictSummary start
   2002-10-18 03:55:06 DistrictSummary end
   2002-10-18 03:55:24 Ajp13Processor[8085][21] Starting background 
thread
   2002-10-18 03:55:24 NationalSummary start
   2002-10-18 03:55:24 NationalSummary end
   2002-10-18 03:55:34 DistrictSummary start
   2002-10-18 03:55:34 DistrictSummary end
   2002-10-18 03:55:41 Page start 206
   2002-10-18 03:55:41 Page end 206
   2002-10-18 03:56:52 Ajp13Processor[8085][22] Starting background 
thread
   2002-10-18 03:56:52 Page start 206
   2002-10-18 03:56:52 Page end 206
   2002-10-18 03:58:16 Page start 206
   2002-10-18 03:58:16 Page end 206
   2002-10-18 03:59:31 Page start 206
   2002-10-18 03:59:31 Page end 206
   2002-10-18 04:00:24 Page start 206
   2002-10-18 04:00:24 Page end 206
   2002-10-18 04:01:34 Page start 206
   2002-10-18 04:01:34 Page end 206




--
To unsubscribe, e-mail:   

For additional commands, e-mail: 





--
To unsubscribe, e-mail:   

For additional commands, e-mail: 

Re: Ajp13Processor starting background threads under low load

[Glenn Nielsen]

Once an Apache process opens a socket connection to Tomcat for Ajp that
connection remains open until that httpd process dies.  That way the
httpd process can reuse that connection.  For Apache 1.3 there is a
one to one mapping between apache httpd processes and Tomcat Ajp13Processors.
You can end up with as many Ajp13Processors as you have httpd processes.
So the MaxProcessors config needs to be greater than the max number
of httpd processes used by apache.  And each Ajp13Processor runs in
its own thread waiting for a request from Apache.

Regards,

Glenn



Lindsay Patten wrote:

Thanks for the pointer Glenn, it was much appreciated!

So, I've been doing thread stacktrace dumps, but what I found doesn't 
make a lot of sense to me.
After restarting tomcat there were 5 processor threads all blocked on a 
monitor:

"Ajp13Processor[8085][0]" daemon prio=1 tid=0x0x4f801b28 nid=0x45ae 
waiting on m
onitor [bd9ff000..bd9ff8ac]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x446602c0> (a org.apache.ajp.tomcat4.Ajp13Processor)
   at java.lang.Object.wait(Object.java:426)
   at 
org.apache.ajp.tomcat4.Ajp13Processor.await(Ajp13Processor.java:305)
   - locked <0x446602c0> (a org.apache.ajp.tomcat4.Ajp13Processor)
   at 
org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:490)
   at java.lang.Thread.run(Thread.java:536)

I presume these are waiting for the scheduler thread to pass them a 
request.

A while later a bunch of background threads get spawned and I do another 
dump/trace,
now there are a whole bunch of threads in a runnable state trying to 
read request packets:

"Ajp13Processor[8085][21]" daemon prio=1 tid=0x0x8289b38 nid=0x70d6 
runnable [ba
fff000..bafff8ac]
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:116)
   at org.apache.ajp.Ajp13.readN(Ajp13.java:429)
   at org.apache.ajp.Ajp13.receive(Ajp13.java:469)
   at org.apache.ajp.Ajp13.receiveNextRequest(Ajp13.java:274)
   at 
org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:371
)
   at 
org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:495)
   at java.lang.Thread.run(Thread.java:536)

Despite the name receiveNextRequest, it looks like that method is being 
called to receive each request including the first. So, the ajp13 
processor is sitting blocked waiting for a full ajp13 packet from apache 
and apache is not sending data and not closing the connection/socket. 
 From the comments in the code it sounds like apache restarts 
periodically, closing the sockets and unblocking the read calls, at 
which point these processors return to the pool - so if apache restarts 
before you reach your threads limit you are ok.

What I don't understand is how apache can be starting up ajp13 
connections to tomcat and then not sending a full packet without this 
causing user visible failures. Shouldn't there be users getting page 
loads timing out on them?  That doesn't seem to be happening.

Does this indicate a fault in apache or is it normal for apache to send 
partial requests without closing the connection?

Any further guidance?

Thanks,
   Lindsay


Glenn Nielsen wrote:

One way to start debugging this type of problem is to tell the java
process running Tomcat to do a Thread Stacktrace Dump.

kill -QUIT 

Then analyze the stack traces for all threads.

Regards,

Glenn


Lindsay Patten wrote:


Hi,
I thought I would take a different tack on my problem with the 
Ajp13Processor using up all of its threads problem.
Does anyone know if the Ajp13Processor has a timeout when it is 
looking for a worker to handle a request? If the worker threads were 
all swapped out and taking a long time to get going and respond, 
would the main thread start a new background thread instead of 
waiting? Or does it just have a list of available workers and blocks 
until the worker it selects responds?
If it is the former that would explain my problem since I have 
experienced my processes getting swapped out and taking a long time 
to respond. The question would then become: is a way to specify how 
long the scheduler should wait before starting a new thread? At least 
I would have something I could take to my service provider. On the 
other hand if I knew for sure that the scheduler thread didn't have a 
timeout on worker threads than I could concentrate on looking 
elsewhere.  Does anyone know? Or know where to look or ask short of 
reading the source?

Thanks,
   Lindsay

Lindsay Patten wrote:

Hi,
I am having a problem where tomcat keeps intermitantly starting new 
ajp13 processor threads, eventually it reaches the max and starts 
refusing connections. My hosting provider (linux machine) provides a 
private version of tomcat for each user with a single apache web 
server, there are typically a large number of processes on the 
machine (>4000). My particular tomcat server (v4.0.3) is not being 
loaded

Re: Ajp13Processor starting background threads under low load

[Lindsay Patten]

Thanks again Glenn!

Now I've got something concrete to take to my service provider for 
resolution.

Cheers,
   Lindsay

Glenn Nielsen wrote:

Once an Apache process opens a socket connection to Tomcat for Ajp that
connection remains open until that httpd process dies.  That way the
httpd process can reuse that connection.  For Apache 1.3 there is a
one to one mapping between apache httpd processes and Tomcat 
Ajp13Processors.
You can end up with as many Ajp13Processors as you have httpd processes.
So the MaxProcessors config needs to be greater than the max number
of httpd processes used by apache.  And each Ajp13Processor runs in
its own thread waiting for a request from Apache.

Regards,

Glenn



Lindsay Patten wrote:

Thanks for the pointer Glenn, it was much appreciated!

So, I've been doing thread stacktrace dumps, but what I found doesn't 
make a lot of sense to me.
After restarting tomcat there were 5 processor threads all blocked on 
a monitor:

"Ajp13Processor[8085][0]" daemon prio=1 tid=0x0x4f801b28 nid=0x45ae 
waiting on m
onitor [bd9ff000..bd9ff8ac]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x446602c0> (a 
org.apache.ajp.tomcat4.Ajp13Processor)
   at java.lang.Object.wait(Object.java:426)
   at 
org.apache.ajp.tomcat4.Ajp13Processor.await(Ajp13Processor.java:305)
   - locked <0x446602c0> (a org.apache.ajp.tomcat4.Ajp13Processor)
   at 
org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:490)
   at java.lang.Thread.run(Thread.java:536)

I presume these are waiting for the scheduler thread to pass them a 
request.

A while later a bunch of background threads get spawned and I do 
another dump/trace,
now there are a whole bunch of threads in a runnable state trying to 
read request packets:

"Ajp13Processor[8085][21]" daemon prio=1 tid=0x0x8289b38 nid=0x70d6 
runnable [ba
fff000..bafff8ac]
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:116)
   at org.apache.ajp.Ajp13.readN(Ajp13.java:429)
   at org.apache.ajp.Ajp13.receive(Ajp13.java:469)
   at org.apache.ajp.Ajp13.receiveNextRequest(Ajp13.java:274)
   at 
org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:371
)
   at 
org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:495)
   at java.lang.Thread.run(Thread.java:536)

Despite the name receiveNextRequest, it looks like that method is 
being called to receive each request including the first. So, the 
ajp13 processor is sitting blocked waiting for a full ajp13 packet 
from apache and apache is not sending data and not closing the 
connection/socket.  From the comments in the code it sounds like 
apache restarts periodically, closing the sockets and unblocking the 
read calls, at which point these processors return to the pool - so 
if apache restarts before you reach your threads limit you are ok.

What I don't understand is how apache can be starting up ajp13 
connections to tomcat and then not sending a full packet without this 
causing user visible failures. Shouldn't there be users getting page 
loads timing out on them?  That doesn't seem to be happening.

Does this indicate a fault in apache or is it normal for apache to 
send partial requests without closing the connection?

Any further guidance?

Thanks,
   Lindsay


Glenn Nielsen wrote:

One way to start debugging this type of problem is to tell the java
process running Tomcat to do a Thread Stacktrace Dump.

kill -QUIT 

Then analyze the stack traces for all threads.

Regards,

Glenn


Lindsay Patten wrote:


Hi,
I thought I would take a different tack on my problem with the 
Ajp13Processor using up all of its threads problem.
Does anyone know if the Ajp13Processor has a timeout when it is 
looking for a worker to handle a request? If the worker threads 
were all swapped out and taking a long time to get going and 
respond, would the main thread start a new background thread 
instead of waiting? Or does it just have a list of available 
workers and blocks until the worker it selects responds?
If it is the former that would explain my problem since I have 
experienced my processes getting swapped out and taking a long time 
to respond. The question would then become: is a way to specify how 
long the scheduler should wait before starting a new thread? At 
least I would have something I could take to my service provider. 
On the other hand if I knew for sure that the scheduler thread 
didn't have a timeout on worker threads than I could concentrate on 
looking elsewhere.  Does anyone know? Or know where to look or ask 
short of reading the source?

Thanks,
   Lindsay

Lindsay Patten wrote:

Hi,
I am having a problem where tomcat keeps intermitantly starting 
new ajp13 processor threads, eventually it reaches the max and 
starts refusing connections. My hosting provider (linux machine) 
provides a private version of tomcat for each user with a single 
apa

Re: Ajp13Processor starting background threads under low load

[Lindsay Patten]

Thanks for the pointer Glenn, it was much appreciated!

So, I've been doing thread stacktrace dumps, but what I found doesn't 
make a lot of sense to me.
After restarting tomcat there were 5 processor threads all blocked on a 
monitor:

"Ajp13Processor[8085][0]" daemon prio=1 tid=0x0x4f801b28 nid=0x45ae 
waiting on m
onitor [bd9ff000..bd9ff8ac]
   at java.lang.Object.wait(Native Method)
   - waiting on <0x446602c0> (a org.apache.ajp.tomcat4.Ajp13Processor)
   at java.lang.Object.wait(Object.java:426)
   at 
org.apache.ajp.tomcat4.Ajp13Processor.await(Ajp13Processor.java:305)
   - locked <0x446602c0> (a org.apache.ajp.tomcat4.Ajp13Processor)
   at 
org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:490)
   at java.lang.Thread.run(Thread.java:536)

I presume these are waiting for the scheduler thread to pass them a request.

A while later a bunch of background threads get spawned and I do another 
dump/trace,
now there are a whole bunch of threads in a runnable state trying to 
read request packets:

"Ajp13Processor[8085][21]" daemon prio=1 tid=0x0x8289b38 nid=0x70d6 
runnable [ba
fff000..bafff8ac]
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:116)
   at org.apache.ajp.Ajp13.readN(Ajp13.java:429)
   at org.apache.ajp.Ajp13.receive(Ajp13.java:469)
   at org.apache.ajp.Ajp13.receiveNextRequest(Ajp13.java:274)
   at 
org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:371
)
   at 
org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:495)
   at java.lang.Thread.run(Thread.java:536)

Despite the name receiveNextRequest, it looks like that method is being 
called to receive each request including the first. So, the ajp13 
processor is sitting blocked waiting for a full ajp13 packet from apache 
and apache is not sending data and not closing the connection/socket. 
From the comments in the code it sounds like apache restarts 
periodically, closing the sockets and unblocking the read calls, at 
which point these processors return to the pool - so if apache restarts 
before you reach your threads limit you are ok.

What I don't understand is how apache can be starting up ajp13 
connections to tomcat and then not sending a full packet without this 
causing user visible failures. Shouldn't there be users getting page 
loads timing out on them?  That doesn't seem to be happening.

Does this indicate a fault in apache or is it normal for apache to send 
partial requests without closing the connection?

Any further guidance?

Thanks,
   Lindsay


Glenn Nielsen wrote:

One way to start debugging this type of problem is to tell the java
process running Tomcat to do a Thread Stacktrace Dump.

kill -QUIT 

Then analyze the stack traces for all threads.

Regards,

Glenn


Lindsay Patten wrote:


Hi,
I thought I would take a different tack on my problem with the 
Ajp13Processor using up all of its threads problem.
Does anyone know if the Ajp13Processor has a timeout when it is 
looking for a worker to handle a request? If the worker threads were 
all swapped out and taking a long time to get going and respond, 
would the main thread start a new background thread instead of 
waiting? Or does it just have a list of available workers and blocks 
until the worker it selects responds?
If it is the former that would explain my problem since I have 
experienced my processes getting swapped out and taking a long time 
to respond. The question would then become: is a way to specify how 
long the scheduler should wait before starting a new thread? At least 
I would have something I could take to my service provider. On the 
other hand if I knew for sure that the scheduler thread didn't have a 
timeout on worker threads than I could concentrate on looking 
elsewhere.  Does anyone know? Or know where to look or ask short of 
reading the source?

Thanks,
   Lindsay

Lindsay Patten wrote:

Hi,
I am having a problem where tomcat keeps intermitantly starting new 
ajp13 processor threads, eventually it reaches the max and starts 
refusing connections. My hosting provider (linux machine) provides a 
private version of tomcat for each user with a single apache web 
server, there are typically a large number of processes on the 
machine (>4000). My particular tomcat server (v4.0.3) is not being 
loaded at all but the ajp13processor threads don't seem to not get 
reused under some circumstances that I don't understand. I have 
added logging statements in my jsp pages and the pages appear to run 
to completion. Each jsp page accesses some info from a mysql db and 
displays it (*Summary) or takes some data from a form and sticks it 
in the db (Page). The system will often run for several hours and 
hundreds of requests without starting any threads, but then 
sometimes, as below, it starts new threads for almost every request 
for a while. Is there a way to determine if the threads are