tomcat 5.0.28 - CPU spikes after heap memory reaches 700MB

2005-08-30 Thread Bhaskar Mulpuri 
We are using tomcat 5.0.28 with apache 2.0 using the jk2 (AJP/1.3)
connector. Apache and tomcat are on different servers, Sun V120's running
Solaris 9. Tomcat server has 3 GB RAM and tomcat JVM is set to 1268Mb
(-Xms1268m -Xmx1268m). The version of mod_jk on the apache web server is
1.2.14.

Jk2 connector is configured as follows

Connector port=3060 maxThreads=200 mainSpareThreads=20
maxSpareThreads=75 enableLookups=false redirectPort=8443 debug=0
protocol=AJP/1.3 /

When the heap memory for the tomcat process reaches 700 Mb (as shown by
solaris top command), for a few requests, the CPU spikes to 90-95 %. This
initially happens for a few requests then gradually progresses to a majority
of the requests. Interestingly, while top command lists the heap usage as
700 Mb, the tomcat manager app shows the amount of free memory between
800-1100 Mb. By this, I am assuming that GC runs periodically, but may not
be releasing it to the O/S. This is the reason why Solaris top command lists
the heap usage as 700 Mb. The heap usage listed by the top command always
increases.  When the CPU spikes up, GC does not seem to be running, since
the only thread that is taking up this CPU is the one processing the request
(as noticed from the prstat -L command). The page that is being displayed is
a jsp page and the size could be around 500 K. The request involves
communication from tomcat to a java process on the same server using sockets
and XML data is transferred and copied to cells on the page. This operation
normally takes up 3-5 % of CPU.

I have noticed that the connections between apache and tomcat are around 19
when this happens. Also once created, the connections persist till tomcat is
restarted. I am not sure if this is good or bad and if this playing any
role. Normally I would think that having the connections open helps since
the sockets do not have to be opened when a request comes in. But I am also
concerned if this is taking up resources on the server.

I have taken a signal dump when the CPU spikes and it shows the 19 threads
reading from a socket and the other 9 threads in a wait condition. 

I have run this under JProbe and it shows the thread processing the request
under normal conditions. It is difficult to reproduce this since we have to
wait till the heap reaches 700 Mb and the application is a custom app
(Workplace) provided by a vendor (FileNet) and I have not been successful in
simulating this with JMeter.

Any ideas on how to find out what is happening when the CPU spikes to 90 %
and why this is happening?

Thanks,
Bhaskar Mulpuri

 



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

RE: tomcat 5.0.28 - CPU spikes after heap memory reaches 700MB

2005-08-30 Thread Caldarale, Charles R 
 From: Bhaskar Mulpuri [mailto:[EMAIL PROTECTED] 
 Subject: tomcat 5.0.28 - CPU spikes after heap memory reaches 700MB
 
 When the heap memory for the tomcat process reaches 700 Mb 
 (as shown by solaris top command), for a few requests, the 
 CPU spikes to 90-95 %.

Is it possible that the application is getting an OOME, catching it, and
simply retrying the request?  This would account for the high CPU
utilization, and GC won't do much if nothing's changed since the last
attempt.  Since OOME covers a variety of resource exhaustions besides
the primary Java heap, you might well be running out of file handles,
filling up the permanent generation, hitting max threads (unlikely), or
???

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

RE: tomcat 5.0.28 - CPU spikes after heap memory reaches 700MB

2005-08-30 Thread Bhaskar Mulpuri 
The heap memory is still listed at 700 Mb when the CPU spikes up. We have
also set the max memory at 2GB for the tomcat JVM process, but we see the
same behavior.

I have not seen any error messages in the logs either. So I have not been
looking at it from the OOME perspective. Also, we checked with the vendor
and they have not mentioned this.

Thanks,
Bhaskar Mulpuri


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]




-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]