Re: [OT] Tomcat Server CPU utilization goes upto 400%
Do as Tim Funk suggested, issue a kill -3 to tomcat when CPU usage gets
high to get an idea of what java Threads are currently doing. Issue a
few kills to get a better view of what part of process takes times. If
it's not enough to find the location of issue, you might want to invest
time in using a profiler to locate problem (attach profiler to running
jvm when it starts to consume CPU).
En l'instant précis du 14/06/07 05:17, Sumit Gaikaiwari s'exprimait en
ces termes:
There is an application deployed that refreshes after every 30 seconds.
After every 30 seconds, it requeries database and fetches the data. This
was a design requirement for this application. Generally large numbers
of sessions of this application are running (450-500).
Regards,
Sumit Gaikaiwari
-Original Message-
From: Andre Prasetya [mailto:[EMAIL PROTECTED]
Sent: Wednesday, June 13, 2007 4:22 PM
To: Tomcat Users List
Subject: Re: Tomcat Server CPU utilization goes upto 400%
i think its more to application problem instead of tomcat problem. Is
there
any application hosted inside tomcat that get infinite loop or use a
kind of
loop checking something that utilize the thread fully ? try solving that
with the application developer
On 6/13/07, Sumit Gaikaiwari [EMAIL PROTECTED] wrote:
Hello,
We are using tomcat 5.0.25 on Red Hat Enterprise Linux server. There
two
applications in webapps. We are facing an issue from around 3-4 months
where the CPU utilization by java process (tomcat) goes quite high:
around 400% in every 4-5 days. The application gets very slow and
tomcat
needs to be restarted for normal operation. The output of top command
is
as below-
[EMAIL PROTECTED] jakarta-tomcat-5.0.25]# top
Tasks: 78 total, 1 running, 77 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.1% us, 0.1% sy, 0.0% ni, 74.5% id, 0.2% wa, 0.0% hi,
0.0% si
Mem: 4086472k total, 2487676k used, 1598796k free, 217464k
buffers
Swap: 6094824k total,0k used, 6094824k free, 1811576k
cached
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
1038 root 16 0 545m 244m 61m S 101 6.1 1156:22 java
1 root 16 0 2556 552 472 S0 0.0 0:01.17 init
2 root RT 0 000 S0 0.0 0:00.31
migration/0
Here is the status of active threads-
Thread Status-
[EMAIL PROTECTED] jakarta-tomcat-5.0.25]# ps -Le -o
pid,user,s,lwp,pcpu,args | awk '$3 != S { print }'
1038 root R 1093 19.8 /usr/local/j2sdk1.4.2_12/bin/java -Xmx256m
-Xms128m -Djava.awt.headless=true -Dsun.awt.font.advancecache=off
-Djava.endorsed.dirs=/usr/local/jakarta-tomcat-5.0.25/common/endorsed
-classpath
/usr/local/j2sdk1.4.2_12/lib/tools.jar:/usr/local/jakarta-tomcat-5.0.25/
bin/bootstrap.jar:/usr/local/jakarta-tomcat-5.0.25/bin/commons-logging-a
pi.jar -Dcatalina.base=/usr/local/jakarta-tomcat-5.0.25
-Dcatalina.home=/usr/local/jakarta-tomcat-5.0.25
-Djava.io.tmpdir=/usr/local/jakarta-tomcat-5.0.25/temp
org.apache.catalina.startup.Bootstrap start
20544 root R 20544 0.0 ps -Le -o pid,user,s,lwp,pcpu,args
This was status at certain instant of time. After 5 hours from this
instant, the utilization reached to 400% and also there were many
threads which were using almost 150% of CPU time.
The system configuration is -
[EMAIL PROTECTED] ~]# wget --quiet -O - --user admin --password nimda
http://localhost:8080/manager/serverinfo
OK - Server info
Tomcat Version: Apache Tomcat/5.0.25
OS Name: Linux
OS Version: 2.6.9-42.0.2.ELsmp
OS Architecture: i386
JVM Version: 1.4.2_12-b03
JVM Vendor: Sun Microsystems Inc.
Please can any one tell me possible cause for this issue? This issue
has
become very critical and needs to be fixed out soon.
Regards,
Sumit Gaikaiwari
-**Nihilent***
*** All information contained in this communication is confidential,
proprietary, privileged and is intended for the addressees only. If
you have
received this E-mail in error please notify mail administrator by
telephone
on +91-20-39846100 or E-mail the sender by replying to this message,
and
then delete this E-mail and other copies of it from your computer
system.
Any unauthorized dissemination, publication, transfer or use of the
contents
of this communication, with or without modifications is punishable
under the
relevant law.
Nihilent has scanned this mail with current virus checking
technologies.
However, Nihilent makes no representations or warranties to the effect
that
this communication is virus-free.
Nihilent reserves the right to monitor all E-mail communications
through its Corporate Network. ***
*-
Re:[OT] Tomcat Server CPU utilization goes upto 400%
Sumit,
Since this is not a Tomcat issue I tagged it as OT.
Is the data query different for each user?
If no, setup a separate thread on startup to do the query and process it as
far as you can. I was able to create a String that was the body of the page,
then each request that came in simply inserted the string in a
document.write on a jsp. It made the response ultra fast.
If yes, look at the data returned. How much are you having to process the
data? I had to iterate through the results of three query's, create objects
and populate the fields from other queries. Then do a sort on the object. It
was very processor intensive. Is there anything you can do with a stored
procedure on the database side. You need to limit the amount of work done on
the data.
Doug
- Original Message -
From: Sumit Gaikaiwari [EMAIL PROTECTED]
To: Tomcat Users List users@tomcat.apache.org
Sent: Thursday, June 14, 2007 7:15 AM
Subject: RE: Tomcat Server CPU utilization goes upto 400%
Hi Doug,
The requirements of this application were as mentioned in the previous
mail. Can I have any work around on the issue? What things you tried
when you had same situation?
Regards,
Sumit Gaikaiwari
-Original Message-
From: PTS [mailto:[EMAIL PROTECTED]
Sent: Thursday, June 14, 2007 9:23 AM
To: Tomcat Users List
Subject: Re: Tomcat Server CPU utilization goes upto 400%
Sumit,
Look at the design of the app. Is each session doing a query every 30
seconds? Also is the database on the same server?
I had an app that was originally designed to be for a few users that
each
session hit the database. Due to the amount of processing of the data it
had
a snowball effect that after more than a few users connected the system
began to fall behind and the system came to a crawl. Since my query was
the
same for all users, I was able to spawn a thread to do a query every few
minutes and cache the results, then each session grabbed a copy of the
cached data from memory.
I would also have to agree that it appears to be an application issue,
especially after experiencing it myself. Or more accurately doing it to
myself.
Doug
- Original Message -
From: Sumit Gaikaiwari [EMAIL PROTECTED]
To: Tomcat Users List users@tomcat.apache.org
Sent: Wednesday, June 13, 2007 11:17 PM
Subject: RE: Tomcat Server CPU utilization goes upto 400%
There is an application deployed that refreshes after every 30 seconds.
After every 30 seconds, it requeries database and fetches the data. This
was a design requirement for this application. Generally large numbers
of sessions of this application are running (450-500).
Regards,
Sumit Gaikaiwari
-Original Message-
From: Andre Prasetya [mailto:[EMAIL PROTECTED]
Sent: Wednesday, June 13, 2007 4:22 PM
To: Tomcat Users List
Subject: Re: Tomcat Server CPU utilization goes upto 400%
i think its more to application problem instead of tomcat problem. Is
there
any application hosted inside tomcat that get infinite loop or use a
kind of
loop checking something that utilize the thread fully ? try solving that
with the application developer
On 6/13/07, Sumit Gaikaiwari [EMAIL PROTECTED] wrote:
Hello,
We are using tomcat 5.0.25 on Red Hat Enterprise Linux server. There
two
applications in webapps. We are facing an issue from around 3-4 months
where the CPU utilization by java process (tomcat) goes quite high:
around 400% in every 4-5 days. The application gets very slow and
tomcat
needs to be restarted for normal operation. The output of top command
is
as below-
[EMAIL PROTECTED] jakarta-tomcat-5.0.25]# top
Tasks: 78 total, 1 running, 77 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.1% us, 0.1% sy, 0.0% ni, 74.5% id, 0.2% wa, 0.0% hi,
0.0% si
Mem: 4086472k total, 2487676k used, 1598796k free, 217464k
buffers
Swap: 6094824k total,0k used, 6094824k free, 1811576k
cached
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
1038 root 16 0 545m 244m 61m S 101 6.1 1156:22 java
1 root 16 0 2556 552 472 S0 0.0 0:01.17 init
2 root RT 0 000 S0 0.0 0:00.31
migration/0
Here is the status of active threads-
Thread Status-
[EMAIL PROTECTED] jakarta-tomcat-5.0.25]# ps -Le -o
pid,user,s,lwp,pcpu,args | awk '$3 != S { print }'
1038 root R 1093 19.8 /usr/local/j2sdk1.4.2_12/bin/java -Xmx256m
-Xms128m -Djava.awt.headless=true -Dsun.awt.font.advancecache=off
-Djava.endorsed.dirs=/usr/local/jakarta-tomcat-5.0.25/common/endorsed
-classpath
/usr/local/j2sdk1.4.2_12/lib/tools.jar:/usr/local/jakarta-tomcat-5.0.25/
bin/bootstrap.jar:/usr/local/jakarta-tomcat-5.0.25/bin/commons-logging-a
pi.jar -Dcatalina.base=/usr/local/jakarta-tomcat-5.0.25
-Dcatalina.home=/usr/local/jakarta-tomcat-5.0.25
-Djava.io.tmpdir=/usr/local/jakarta-tomcat-5.0.25/temp
org.apache.catalina.startup.Bootstrap start
20544 root R 20544 0.0 ps -Le -o pid,user,s,lwp,pcpu,args
