ThreadPool safety checks

2001-05-24 Thread Hector Gonzalez

Hi all,

After performing stress testing on Tomcat 3.2.1 running on Linux, I found
that performance would always decrease over time until the server stopped
responding completely. I was accessing tomcat as an stand alone server on
port 8080 and requesting a servlet that returned a null body (null servlet),
tomcat was running on IBM 1.3 JVM, using native threads. The clients where
three machines each running 20 threads requesting the null servlet.
Initially tomcat would handle about 400 requests/sec, but after about 20
minutes requests per second would be down to about 20, and thereafter tomcat
would completely stop.

After looking at the code, I found out that all the threads in the thread
pool of tcp connections were busy and none would become idle. The attached
TcpWorkerThreads would not return from processing the request, I don't
really know why. I modified ThreadPool.java to perform two sanity checks on
the pool:

1. If a thread has been processing a request for too long (as specified in a
parameter) the thread is stoped.
2. After a thread has already processed X requests or more (X
TcpWorkerThreads have been attached), it is terminated and a new thread is
created.

The checks are performed by the Monitor thread, when it calls
checkSpareControllers.

After implementing these changes I ran the stress tests again with very good
results: performance remains at around 400 requests/sec indefinetly.

The safety checks can be completely disabled through configuration
parameters if desired.

As I am new to this list, I am not sure what the procedure to review and
possibly incorporate my changes into the code are. I modified three files:
org.apache.tomcat.util.ThreadPool, org.apache.tomcat.util.ThreadPoolRunnable
and org.apache.tomcat.service.PoolTcpEndPoint.java. I am attaching the
modified files. There is a lot of debugging info in those, I could clean
them up if you decide that it is worth including the changes in the main
branch of Tomcat.

Regards

Hector Gonzalez
QuestionExchange - http://www.questionexchange.com
[EMAIL PROTECTED]
Voice: (617)451-1527 Fax: (617)451-1487

 ThreadPool.java
 ThreadPoolRunnable.java
 PoolTcpEndpoint.java


[PATCH] ThreadPool rewrite with sanity checks

2001-06-01 Thread Hector Gonzalez

Hi all,

After performing stress testing on Tomcat 3.2.2 I found that one area of
trouble was the Thread Pool. Long running servlets may eat up all resources
and
prevent tomcat from accepting any new connections on the socket. Also for
some still unknown reason performance would decrease over time.

I have patched the code to do perform two safety checks:

1. If a thread has been processing a request for too long (as specified in a
parameter) the thread is stoped.
2. After a thread has already processed X requests or more (X
TcpWorkerThreads have been attached), it is terminated and a new thread is
created.

I also changed the code and moved most synchronization from ThreadPool to
ObjectHASH (a fully synchronized hash table)

I tested the patched code with 60 threads from 3 different machines with
very
good results in terms of reliability and performance.

This patch involves changes to:

util/ThreadPool.java
util/ThreadPoolRunnable.java
service/PoolTcpEndPoint.java

Regards
Hector Gonzalez
[EMAIL PROTECTED]


Index: ThreadPool.java
===
RCS file: 
/home/cvspublic/jakarta-tomcat/src/share/org/apache/tomcat/util/Attic/ThreadPool.java,v
retrieving revision 1.9.2.2
diff -u -r1.9.2.2 ThreadPool.java
--- ThreadPool.java 2001/04/23 02:16:03 1.9.2.2
+++ ThreadPool.java 2001/06/01 20:10:21
@@ -1,7 +1,7 @@
 /*
- * $Header: 
/home/cvspublic/jakarta-tomcat/src/share/org/apache/tomcat/util/Attic/ThreadPool.java,v
 1.9.2.2 2001/04/23 02:16:03 marcsaeg Exp $
- * $Revision: 1.9.2.2 $
- * $Date: 2001/04/23 02:16:03 $
+ * $Header: 
+/home/cvs/jakarta-tomcat/src/share/org/apache/tomcat/util/Attic/ThreadPool.java,v 
+1.9.2.1 2000/07/06 22:20:17 alex Exp $
+ * $Revision: 1.9.2.1 $
+ * $Date: 2000/07/06 22:20:17 $
  *
  * 
  *
@@ -71,12 +71,17 @@
 import org.apache.tomcat.logging.*; 
 
 /**
- * A thread pool that is trying to copy the apache process management.
+ * A thread pool that creates a number of threads and
+ * assigns them to runnable objects as requested.
+ * The pool provides maintenance on the individual
+ * threads by cheking running time and number of runs.
+ * Tried to keep synchronization to a minimum, most of the
+ * synchronization happens at the idleWorkers (ObjectHASH)
+ * level.
  *
- * @author Gal Shachor
+ * @author Hector Gonzalez
  */
-public class ThreadPool  {
-
+public class ThreadPool{
 /*
  * Default values ...
  */
@@ -84,470 +89,675 @@
 public static final int MAX_SPARE_THREADS = 50;
 public static final int MIN_SPARE_THREADS = 10;
 public static final int WORK_WAIT_TIMEOUT = 60*1000;
+public static final int MAX_THREAD_RUNS = 100;
+public static final int MAX_THREAD_ITERATIONS = 10;
 
-/*
- * Where the threads are held.
- */
-protected Vector pool;
+static int debug=0;
+LogHelper loghelper = new LogHelper("tc_log", "ThreadPool");
 
 /*
- * A monitor thread that monitors the pool for idel threads.
+ * Max number of threads that you can open in the pool.
  */
-protected MonitorRunnable monitor;
-
+protected int maxThreads;
 
 /*
- * Max number of threads that you can open in the pool.
+ * Min number of idle threads that you can leave in the pool.
  */
-protected int maxThreads;
+protected int maxSpareThreads;
 
 /*
- * Min number of idel threads that you can leave in the pool.
+ * Max number of idle threads that you can leave in the pool.
  */
 protected int minSpareThreads;
 
 /*
- * Max number of idel threads that you can leave in the pool.
+ * Number of runs before thread is replaced
  */
-protected int maxSpareThreads;
+protected int maxThreadRuns;
 
-/*
- * Number of threads in the pool.
+/* 
+ * Number of times the monitor thread will run before
+ * stopping a thread that has been continuously busy
+ * effective time before stopping = WORK_WAIT_TIMEOUT * MAX_THREAD_ITERATIONS
  */
-protected int currentThreadCount;
+protected int maxThreadIterations;
 
 /*
- * Number of busy threads in the pool.
+ * Keep list of idle threads
  */
-protected int currentThreadsBusy;
+ObjectHASH idleWorkers;
 
 /*
- * Flag that the pool should terminate all the threads and stop.
+ * List of all the created threads, idle and busy
  */
-protected boolean stopThePool;
-
-static int debug=0;
+Hashtable workerList;
 
-/**
- * Helper object for logging
- **/
-LogHelper loghelper = new LogHelper("tc_log", "ThreadPool");
-
-public ThreadPool() {
-maxThreads  = MAX_THREADS;
-maxSpareThreads = MAX_SPARE_THREADS;
-minSpareThreads = MIN_SPARE_THREADS;
-currentThreadCount  = 0;
-currentThreadsBusy  = 0;
-stopThePool = false;
-