Tomcat Acceptor Thread goes into wait() to never come back - Tomcat 6.0
Hi,
I believe I've found a race condition in Tomcat that causes the http port to
be non-responsive. It exists in 6.0 and also in 5.5 (although the code has
been refactored).
I could not find any reference to it in the Bug database or the mailing list
archives.
Consider a tomcat instance with maxThreads set to 2, i.e. you have 2 tomcat
threads to service incoming requests.
The sequence of events is as follows:
1. Thread 1 and Thread 2 are both servicing a request each.
2. A third request comes in.
3. In class JIOEndpoint.java, the acceptor thread calls methods
processSocket() which then calls getWorkerThread() which then calls
createWorkerThread().
4. createWorkerThread() returns null since both threads are busy processing
the two requests.
5. Here is the race condition in method getWorkerThread() in the code shown
below
protected Worker getWorkerThread(){
...
Worker workerThread = createWorkerThread();
while (workerThread == null) {
try {
synchronized (workers) {
workers.wait();
}
}
...
}
The acceptor thread executes the while(workerThread == null) statement and
is then switched out by the CPU.
The two threads executing the two requests complete and go into
Worker.await() waiting for the next job after executing method
recycleWorkerThread().
The acceptor thread is switched back into CPU and executes the synchronized
block and goes into the wait().
At this point, there aren't any Worker threads out there processing requests
and therefore there isn't any thread to wake up the acceptor thread.
The application is non-responsive after this.
Thoughts?
A simple solution would be to check if curThreadsBusy 0 in the
synchronized block before going into wait() in method getWorkerThread().
Thanks,
Harshad
Stack Traces below:
bda19102143 id=1578 in WAITING on
lock=org.apache.tomcat.util.net.jioendpoint$wor...@13aa4ee3^m
at java.lang.Object.wait(Native Method)^M
at java.lang.Object.wait(Object.java:485)^M
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)^M
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)^M
at java.lang.Thread.run(Thread.java:619)^M
http-8091-Acceptor-0 id=43 in WAITING on
lock=org.apache.tomcat.util.net.jioendpoint$workerst...@13bd7b6a^m
at java.lang.Object.wait(Native Method)^M
at java.lang.Object.wait(Object.java:485)^M
at
org.apache.tomcat.util.net.JIoEndpoint.getWorkerThread(JIoEndpoint.java:700)^M
at
org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:731)^M
at
org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:313)^M
at java.lang.Thread.run(Thread.java:619)^M
Re: Tomcat Acceptor Thread goes into wait() to never come back - Tomcat 6.0
Hi,
If the threads are in the race condition, can it be any possibility
the Tomcat will become responsive again - say if the current
processing request thread is dropped due to request timeout and the
browser is closed?
On Tue, Mar 2, 2010 at 10:27 AM, Harshad Kamat hnka...@gmail.com wrote:
Hi,
I believe I've found a race condition in Tomcat that causes the http port to
be non-responsive. It exists in 6.0 and also in 5.5 (although the code has
been refactored).
I could not find any reference to it in the Bug database or the mailing list
archives.
Consider a tomcat instance with maxThreads set to 2, i.e. you have 2 tomcat
threads to service incoming requests.
The sequence of events is as follows:
1. Thread 1 and Thread 2 are both servicing a request each.
2. A third request comes in.
3. In class JIOEndpoint.java, the acceptor thread calls methods
processSocket() which then calls getWorkerThread() which then calls
createWorkerThread().
4. createWorkerThread() returns null since both threads are busy processing
the two requests.
5. Here is the race condition in method getWorkerThread() in the code shown
below
protected Worker getWorkerThread(){
...
Worker workerThread = createWorkerThread();
while (workerThread == null) {
try {
synchronized (workers) {
workers.wait();
}
}
...
}
The acceptor thread executes the while(workerThread == null) statement and
is then switched out by the CPU.
The two threads executing the two requests complete and go into
Worker.await() waiting for the next job after executing method
recycleWorkerThread().
The acceptor thread is switched back into CPU and executes the synchronized
block and goes into the wait().
At this point, there aren't any Worker threads out there processing requests
and therefore there isn't any thread to wake up the acceptor thread.
The application is non-responsive after this.
Thoughts?
A simple solution would be to check if curThreadsBusy 0 in the
synchronized block before going into wait() in method getWorkerThread().
Thanks,
Harshad
Stack Traces below:
bda19102143 id=1578 in WAITING on
lock=org.apache.tomcat.util.net.jioendpoint$wor...@13aa4ee3^m
at java.lang.Object.wait(Native Method)^M
at java.lang.Object.wait(Object.java:485)^M
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)^M
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)^M
at java.lang.Thread.run(Thread.java:619)^M
http-8091-Acceptor-0 id=43 in WAITING on
lock=org.apache.tomcat.util.net.jioendpoint$workerst...@13bd7b6a^m
at java.lang.Object.wait(Native Method)^M
at java.lang.Object.wait(Object.java:485)^M
at
org.apache.tomcat.util.net.JIoEndpoint.getWorkerThread(JIoEndpoint.java:700)^M
at
org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:731)^M
at
org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:313)^M
at java.lang.Thread.run(Thread.java:619)^M
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Acceptor Thread goes into wait() to never come back - Tomcat 6.0
From: Harshad Kamat [mailto:hnka...@gmail.com]
Subject: Tomcat Acceptor Thread goes into wait() to never come back -
Tomcat 6.0
I believe I've found a race condition in Tomcat that causes the http
port to be non-responsive.
I believe you are correct. To close the timing window, the lock needs to
remain up across the call to createWorkerThread() and the test for null.
Something like this would work:
protected Worker getWorkerThread() {
Worker workerThread;
// Allocate a new worker thread
synchronized (workers) {
while ((workerThread = createWorkerThread()) == null) {
try {
workers.wait();
} catch (InterruptedException e) {
// Ignore
}
}
}
return workerThread;
}
While we're at it, the synchronized clause in createWorkerThread() can now be
eliminated, and the method simplified (and a lot of useless and confusing
parentheses removed):
protected Worker createWorkerThread() {
if (workers.size() 0) {
curThreadsBusy++;
return workers.pop();
}
if (maxThreads 0 || curThreads maxThreads) {
curThreadsBusy++;
if (curThreadsBusy == maxThreads) {
log.info(sm.getString(endpoint.info.maxThreads,
Integer.toString(maxThreads), address,
Integer.toString(port)));
}
return newWorkerThread();
}
return null;
}
The two methods could be combined, but it would probably be uglier. One could
also fix the symptom by putting a time limit on the wait() call, but that
offends the sensibilities.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you received
this in error, please contact the sender and delete the e-mail and its
attachments from all computers.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
