I have 3 areas of concern. I posted all in one e-mail to avoid duplicating info and because they might be related (sorry if that was wrong). I had to switch to my Tomcat server Monday because the WebSphere one got hung up (a very rare occurrence). I was up on Tomcat for 2 hours. I had increased maxProcessors to 10,000 since that is how many my other iSeries uses (WebSphere:270 + IBM HTTP Server:10,005). The Tomcat server ran out of memory well before that point (I think the thread count was between 3500 and 4500), which is hard to believe since I start the jvm with these memory options: -Xms1024m and -Xmx3072m (the machine has 4GB of RAM). Here's where I'm at now in each area:
#1 Threads: I ran a dump of the jvm (iSeries equivalent of the unix "quit", I think) about an hour after taking the Tomcat server off of the internet. It has 1773 threads in use. None of the stack traces reference any of my classes. 12 of the threads appear to be used by Tomcat for the other applications (examples, etc.). 86 of them are the open socket connections to the various remote databases my application needs. The other 1675 break down as follows:
-----------------------------------
63 of them are like this:
Thread priority: 5
Thread status: Waiting
Wait object: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable
Thread group: main
Runnable: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable
Stack:
org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run()V+0 (ThreadPool.java:507)
java/lang/Thread.run()V+11 (Thread.java:484)
Locks:
None
(any ideas here?)
-----------------------------------
1610 look like this:
Thread priority:5
Thread status: Java wait
Thread group: main
Runnable: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable
Stack:
java/net/PlainSocketImpl.accept(Ljava/net/SocketImpl;)V+0 (PlainSocketImpl.java:413)
java/net/ServerSocket.implAccept(Ljava/net/Socket;)V+36 (ServerSocket.java:241)
com/sun/net/ssl/internal/ssl/SSLServerSocketImpl® .accept()Ljava/net/Socket;+0 (DashoA6275:0)
org/apache/tomcat/util/net/jsse/JSSESocketFactory.acceptSocket(Ljava/net/ServerSocket;)Ljava/net/Socket;+0 (JSSESocketFactory.java:238)
org/apache/tomcat/util/net/PoolTcpEndpoint.acceptSocket()Ljava/net/Socket;+0 (PoolTcpEndpoint.java:332)
org/apache/tomcat/util/net/TcpWorkerThread.runIt([Ljava/lang/Object;)V+0 (PoolTcpEndpoint.java:492)
org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run()V+0 (ThreadPool.java:507)
java/lang/Thread.run()V+11 (Thread.java:484)
Locks:
None
-----------------------------------
There are two other threads with identical call stacks to the 1610 group. The difference is that they both show a thread status of "Running" . Also, one of them lists the following lock information:
Locks:
Object id: E4777F625DC75890
Number of locks: 1
There is no reference to this object id anywhere else in any of the logs. Does anyone have any ideas about these? Should I be concerned about them, or are these just sockets the system is keeping available because it doesn't "need" to garbage collect them?
#2 - Manager app session count:
Sometime in the first hour (probably at the 30 minute mark) of sitting idle the manager app session count fell from 3671 to 1795. After three days it still returns the following to a /sessions request: "... maximum session inactive interval 30 - <40 minutes:1795 sessions"
- Is there a way to query Tomcat about these sessions via the manager app or my web app?
- What would cause the manager app to not break down the list in 10 minute intervals as sessions near the timeout of 30 minutes (it has not broken down the list in over 6 weeks of testing)?
- There can be a large number of objects references stored in a session. I do call invalidate() as part of the logoff servlet, but I don't null the references. Do I need to null them? If so, do I need to null each reference inside a collection, or just the collection itself?
#3 - Persistent sessions:
I assume that my <Manager> group is correct to eliminate persistent sessions(see below). If not, please let me know.
Thanks again for any help!
Mark Walker
From: Glenn Nielsen <[EMAIL PROTECTED]> Reply-To: "Tomcat Users List" <[EMAIL PROTECTED]> To: Tomcat Users List <[EMAIL PROTECTED]> Subject: Re: Thread count growth, /manager/sessions, and persistence Date: Tue, 26 Nov 2002 22:36:50 -0600Regarding sessions, by default a JSP page creates a session. All your JSP pages which don't require a session should have the attribute session="false" set in the page directive. The number of threads being used is not related to session management. To troubleshoot problems running out of Processors try telling the JVM to do a thread Stackdump when this happens. On unix you send the java process a QUIT signal. Analyze the thread Stackdumps to determine what the bottleneck is for your application running in Tomcat. i.e. What are all the threads doing when you run out of processors. BTW, 2500 processors seems excessive. Does your site really need to handle 2500 concurrent requests? Also try starting the JVM with the -verbose:gc arg using the CATALINA_OPTS environment variable. This will generate garbage collection data to stdout (catalina.out). Look for very long GC times in the 10's of seconds. You may need to tune your java min/max heap sizes and other GC related options to improve GC performance. When GC happens Tomcat freezes. This can cause requests to build up on the Coyote socket up to your acceptCount limit. And it can cause other failures for things which can timeout such as db connections. Regards, Glenn Mark Walker wrote:I am trying to port an application over to Tomcat v 4.1.12 which has worked fine for two years running in WebSphere on an iSeries. The JSPs were at the .91 level. I have updated them to the 1.2 level. In addition, I have installed the tomcat-util.jar from 4.1.14 to fix the erroneous ssl debug messaging. From the user's perspective everything looks the same. On the server side though, I seem to have a runaway thread problem. The Tomcat manager app shows the number of active sessions to be anywhere between 400 and 700 during the day, dwindling to a few dozen at night. (Incidentally, the session count is always in one group: the 30-40 minute one - I expected a breakdown in 10 minute intervals). But the thread count just keeps rising. When the count reaches the maxProcessors limit a message, "[INFO]ThreadPool - -All threads are busy, waiting. Please increase maxThreads or check the servlet status" appears in the log. I have tried "suggesting" garbage collection to the JVM at various times but to no avail.
Here are the <Connector> definitions in use:
<Connector className="org.apache.coyote.tomcat4.CoyoteConnector"
port="80" minProcessors="5" maxProcessors="2500"
enableLookups="true" redirectPort="443"
acceptCount="25" debug="0" connectionTimeout="20000"
useURIValidationHack="false" />
<Connector className="org.apache.coyote.tomcat4.CoyoteConnector"
port="443" minProcessors="25" maxProcessors="2500"
enableLookups="true"
acceptCount="25" debug="0" scheme="https" secure="true"
useURIValidationHack="false">
<Factory className="org.apache.coyote.tomcat4.CoyoteServerSocketFactory"
clientAuth="false" keystoreFile="**filename**"
keystorePass="**password**" keystoreType="**type**" protocol="TLS"/>
</Connector>
(There is also the standard Coyote/JK2 AJP 1.3 Connector on 8009 but it looks like I don't need that as I'm using Tomcat as a standalone server)
Here are my questions:
Are there any known issues with Tomcat and excessive thread counts?
(Grasping at straws here I know. I have not found any other relevant reference to Tomcat having thread issues like these, yet the same Java code has served well for two years.)
Could the lack of a breakdown list in the /manager/sessions output be a symptom of a thread problem or is it just a newbie configuration error?
Another item that might be an issue is persistent sessions. I don't want them. I invalidate() the session when a user logs out. To that end I have included the following within each <context> group:
<Manager className="org.apache.catalina.session.PersistentManager"
debug="0"
saveOnRestart="false"
maxActiveSessions="-1"
minIdleSwap="1800"
maxIdleSwap="-1"
maxIdleBackup="-1">
<Store className="org.apache.catalina.session.FileStore"/>
</Manager>
(I have also used -1 for the minIdleSwap. I changed it to 1800 to try and keep the sessions in memory in case that was part of the problem.) Is this the right approach to take? I'm concerned that something is wrong here because if I end Tomcat and restart it the cpu usage goes from an average of 6% to 40+%. A second restart puts it up to 99%. The system maintains these rates until I switch to another server (At least the pages still serve fine though).
Lastly, in the event that WebSphere has just been covering my tracks well, I would also welcome suggestions of Java discussion lists anyone has had success with where I could post some particulars regarding our multithreaded processing.
Thanks in advance!
Mark Walker
_________________________________________________________________
Add photos to your e-mail with MSN 8. Get 2 months FREE*. http://join.msn.com/?page=features/featuredemail
--
To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>
--
To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>
_________________________________________________________________
The new MSN 8: advanced junk mail protection and 2 months FREE* http://join.msn.com/?page=features/junkmail
--
To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>
