ThreadPool safety checks

2001-05-24 Thread Hector Gonzalez

Hi all,

After performing stress testing on Tomcat 3.2.1 running on Linux, I found
that performance would always decrease over time until the server stopped
responding completely. I was accessing tomcat as an stand alone server on
port 8080 and requesting a servlet that returned a null body (null servlet),
tomcat was running on IBM 1.3 JVM, using native threads. The clients where
three machines each running 20 threads requesting the null servlet.
Initially tomcat would handle about 400 requests/sec, but after about 20
minutes requests per second would be down to about 20, and thereafter tomcat
would completely stop.

After looking at the code, I found out that all the threads in the thread
pool of tcp connections were busy and none would become idle. The attached
TcpWorkerThreads would not return from processing the request, I don't
really know why. I modified ThreadPool.java to perform two sanity checks on
the pool:

1. If a thread has been processing a request for too long (as specified in a
parameter) the thread is stoped.
2. After a thread has already processed X requests or more (X
TcpWorkerThreads have been attached), it is terminated and a new thread is
created.

The checks are performed by the Monitor thread, when it calls
checkSpareControllers.

After implementing these changes I ran the stress tests again with very good
results: performance remains at around 400 requests/sec indefinetly.

The safety checks can be completely disabled through configuration
parameters if desired.

As I am new to this list, I am not sure what the procedure to review and
possibly incorporate my changes into the code are. I modified three files:
org.apache.tomcat.util.ThreadPool, org.apache.tomcat.util.ThreadPoolRunnable
and org.apache.tomcat.service.PoolTcpEndPoint.java. I am attaching the
modified files. There is a lot of debugging info in those, I could clean
them up if you decide that it is worth including the changes in the main
branch of Tomcat.

Regards

Hector Gonzalez
QuestionExchange - http://www.questionexchange.com
[EMAIL PROTECTED]
Voice: (617)451-1527 Fax: (617)451-1487

 ThreadPool.java
 ThreadPoolRunnable.java
 PoolTcpEndpoint.java


Re: ThreadPool safety checks

2001-05-24 Thread Dan Milstein

Hector,

First off, it looks like you've done some very, very helpful work.  A few
suggestions:

1) For info about how to contribute code, processes, etc, look at:

http://jakarta.apache.org/site/getinvolved.html

And, specifically, about the format for proposed code changes (i.e. patches
instead of full files):

http://jakarta.apache.org/site/source.html

2) You might take a look at 3.2.2 -- it has a lot of bugfixes, and should be
released live very soon.  I don't know if any work was done on this
particular issue, but it would be worth a quick check.  Info about how to
obtain 3.2.2 can be found on the site.

Thanks again,
-Dan

Hector Gonzalez wrote:
 
 Hi all,
 
 After performing stress testing on Tomcat 3.2.1 running on Linux, I found
 that performance would always decrease over time until the server stopped
 responding completely. I was accessing tomcat as an stand alone server on
 port 8080 and requesting a servlet that returned a null body (null servlet),
 tomcat was running on IBM 1.3 JVM, using native threads. The clients where
 three machines each running 20 threads requesting the null servlet.
 Initially tomcat would handle about 400 requests/sec, but after about 20
 minutes requests per second would be down to about 20, and thereafter tomcat
 would completely stop.
 
 After looking at the code, I found out that all the threads in the thread
 pool of tcp connections were busy and none would become idle. The attached
 TcpWorkerThreads would not return from processing the request, I don't
 really know why. I modified ThreadPool.java to perform two sanity checks on
 the pool:
 
 1. If a thread has been processing a request for too long (as specified in a
 parameter) the thread is stoped.
 2. After a thread has already processed X requests or more (X
 TcpWorkerThreads have been attached), it is terminated and a new thread is
 created.
 
 The checks are performed by the Monitor thread, when it calls
 checkSpareControllers.
 
 After implementing these changes I ran the stress tests again with very good
 results: performance remains at around 400 requests/sec indefinetly.
 
 The safety checks can be completely disabled through configuration
 parameters if desired.
 
 As I am new to this list, I am not sure what the procedure to review and
 possibly incorporate my changes into the code are. I modified three files:
 org.apache.tomcat.util.ThreadPool, org.apache.tomcat.util.ThreadPoolRunnable
 and org.apache.tomcat.service.PoolTcpEndPoint.java. I am attaching the
 modified files. There is a lot of debugging info in those, I could clean
 them up if you decide that it is worth including the changes in the main
 branch of Tomcat.
 
 Regards
 
 Hector Gonzalez
 QuestionExchange - http://www.questionexchange.com
 [EMAIL PROTECTED]
 Voice: (617)451-1527 Fax: (617)451-1487
 
   
   Name: ThreadPool.java
ThreadPool.javaType: unspecified type (application/octet-stream)
   Encoding: quoted-printable
 
   Name: ThreadPoolRunnable.java
ThreadPoolRunnable.javaType: unspecified type (application/octet-stream)
   Encoding: quoted-printable
 
Name: PoolTcpEndpoint.java
PoolTcpEndpoint.javaType: unspecified type (application/octet-stream)
Encoding: quoted-printable

-- 

Dan Milstein // [EMAIL PROTECTED]