As is usual - long reply at the end.
On 2/12/2013 11:48 AM, Cool Techi wrote:
Thanks Everyone for the help, I will enable thread dump for my Tomcat, just
reading up on how to get that done. My database connection string is given
below.
<Resource name="jdbc/bm" type="javax.sql.DataSource"
maxActive="250" maxIdle="5" username="root" testWhileIdle="true"
removeAbandonedTimeout="60" maxWait="-1" removeAbandoned="true"
validationQuery="select 1" driverClassName="com.mysql.jdbc.Driver" password="xxxxxxx"
minEvictableIdleTimeMillis="30000" timeBetweenEvictionRunsMillis="300000"
url="jdbc:mysql://localhost:3306/brandmanagement?autoReconnect=true&useUnicode=true&characterEncoding=utf-8"/>
<Resource name="jdbc/s360" type="javax.sql.DataSource"
maxActive="250" maxIdle="5" username="root" testWhileIdle="true"
removeAbandonedTimeout="60" maxWait="-1" removeAbandoned="true"
validationQuery="select 1" driverClassName="com.mysql.jdbc.Driver" password="xxxxxxx"
minEvictableIdleTimeMillis="30000" timeBetweenEvictionRunsMillis="300000"
url="jdbc:mysql://localhost:3306/brandmanagement?autoReconnect=true&useUnicode=true&characterEncoding=utf-8"/>
I use quartz schedular(spring based) to perform some tasks on reqular
intervals, Log4j is used in those methods, so will cross check those methods
again.
Really appreciate your time and help.
Regards,
Ayush
Date: Tue, 12 Feb 2013 11:36:52 -0800
From: its_toas...@yahoo.com
To: users@tomcat.apache.org
Subject: Re: Tomcat hanging unexpectedly [URGENT]
On 2/12/2013 10:38 AM, Pid wrote:
On 12/02/2013 17:53, Cool Techi wrote:
I am new to this, I have just installed YourKit and see a lot of
threads blocked,but currently my response are fine, the blocked
thread are showing the following
Blocker Thread QuartzSchedular_Worker-6 native ID ..
On clicking on the Blocker thread it shows me the following,
org.apache.log4j.Category.callAppenders
com.mysql.jdbc.SingleByteCharsetConverter.getInstance
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrorObject()
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.addObjectToPool()
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.getNumIdle
Does any of these sound like reason. It would be great to get
pointed in the right direction.
Regards,Ayush
Oh dear.
Are you doing your app logging to a database?
Is it the same database that your app uses?
Is it the same connection pool that your app uses?
p
This sounds like a lot of things. However, without borrowing your
crystal ball, I can't tell anything about the Quartz scheduler.
A couple of things:
1. Upgrade your Java to at least the latest 1.6.0_xx version
The one you are using has known server (and client) security issues.
2. Upgrade your Tomcat to at least the latest 6.xx version
There are many improvements and bug fixes.
3. [URGENT]
I understand that this problem is urgent for you. However, the people
here contribute time on a volunteer basis (I'm taking a break from
writing documentation for a new system). We'll try to help as time
allows, but if you need an SLA then you should purchase commercial support.
4. Possible root cause
I'm guessing (based on your partial listing of classes in a thread dump)
is that you either have a logging problem, a database connectivity
problem, or a Quartz scheduler problem.
Or all of the above.
If there was one problem, then I (guessing) would look to see if you're
logging via a Quartz scheduler job through log4j to a MySQL database.
Now that I've got my guess out of the way, here are some pointers.
a. Thread dumps
You said you've taken them, but have only posted snippets. Post a
complete thread dump inline (mailing list strips attachments). For your
own use, take several 3-10 seconds apart to see what's changing.
b. Check your database
It sounds like you have a connection leak. When this happens, how many
active connections do you have to your MySQL database?
c. Connection leak
As noted above, you may have a connections leak. Post your Resource
definition (hopefully it's in context.xml) with passwords and usernames
changed.
While you're at it, read:
http://tomcat.apache.org/tomcat-6.0-doc/jndi-resources-howto.html#JDBC_Data_Sources
In particular, note removeAbandoned and logAbandoned. These will impact
performance, so do it on a test system.
Date: Tue, 12 Feb 2013 15:35:40 -0200 From:
edsonrich...@hotmail.com To: users@tomcat.apache.org Subject: Re:
Tomcat hanging unexpectedly [URGENT]
There is a similar thread published a week ago. It can be caused
by almost anything, and my last suspect would be Tomcat it self.
Most of time, the cause is some programming error, like: -
missing step out of recursive loop - loading too many objects is
memory (for example, when using JPA where graph of objects are
all connected - so, when you load one object, all database is
read into memory) - some deadlock in multi-threaded
application...
How to discover:
1) Try to get a memory dump and check how your memory is being
used
2) Get a heap dump (stack trace, as well) or currently
running application when it is hang, and check for deadlocks
3) Use VisualVM to get a big picture of what the virtual machine is
doing
+ 1 here
Run this on a test environment, load up the system, and do heap dumps
and thread dumps.
4) Put the application running under profiler (like JRockit
Mission Control, or NetBeans profiler)
These are some ideas...
Regards,
Edson
Em 12/02/2013 15:27, Cool Techi escreveu:
Hi,
We are using tomcat 6.0.26 on windows 2008 web server, with Jdk
1.6.32 installed on the server.
The server seems to be running smoothly most of the time, but
suddenly the memory used goes very high and tomcat stops
responding, the logs don't show any out of memory or thread
exceptions.
What could be causing this?
Regards, Ayush
Oh, and please do not top post. I had to bounce back and forth through
the message to figure out what's been said and where to reply.
. . . just my two cents
/mde/
<Resource name="jdbc/bm"
type="javax.sql.DataSource"
maxActive="250"
maxIdle="5"
username="root"
testWhileIdle="true"
removeAbandonedTimeout="60"
maxWait="-1"
removeAbandoned="true"
validationQuery="select 1"
driverClassName="com.mysql.jdbc.Driver"
password="xxxxxxx"
minEvictableIdleTimeMillis="30000"
timeBetweenEvictionRunsMillis="300000"
url="jdbc:mysql://localhost:3306/brandmanagement
?autoReconnect=true
&useUnicode=true
&characterEncoding=utf-8"
/>
<Resource name="jdbc/s360"
type="javax.sql.DataSource"
maxActive="250"
maxIdle="5"
username="root"
testWhileIdle="true"
removeAbandonedTimeout="60"
maxWait="-1"
removeAbandoned="true"
validationQuery="select 1"
driverClassName="com.mysql.jdbc.Driver"
password="xxxxxxx"
minEvictableIdleTimeMillis="30000"
timeBetweenEvictionRunsMillis="300000"
url="jdbc:mysql://localhost:3306/brandmanagement
?autoReconnect=true
&useUnicode=true
&characterEncoding=utf-8"
/>
I broke up the JDBC URL so it would fit inside a mail message more
conveniently.
So, let me understand the configuration a bit:
1. root account
This is not a good thing, because if someone manages to perform an SQL
injection attack, all your (data)bases belong to that person.
2. autoReconnect=true
According to several threads on the MySQL forums, this is no longer
recommended (and has been known to fail). You're using a
validationQuery, and testOnBorrow defaults to true. You also have
testWhileIdle set to true, so I don't see any advantage to
autoReconnect=true in your URL.
3. removeAbandoned=true
After 60 seconds of inactivity a connection is assumed to be inactive
and tossed back into the pool. Hopefully none of your database
transactions last longer than that.
4. maxWait=-1
This means you'll wait forever for a new database connection. If your
database is busy (more about that shortly), then a thread will block
forever while it waits for a database connection (hmmm, sounds like a
symptom).
5. Evictions
From your settings, you check 3 (out of a potential 250!) connections
every 5 minutes. If the thread has been idle for 30 seconds, you remove
the thread and shrink the pool.
6. Connections
There are a total potential of 500 database connections to one URL. This
seems to be a bit excessive, although I don't know the application.
Here's a possible scenario, although I've not thought about it much.
1. There is a Quartz job that talks to MySQL
2. It doesn't close the connection properly
3. autoReconnect makes sure the connection never is idle
4. Maximum number of connections are reached
5. The next thread waiting for a database connection waits forever
6. All new Quartz jobs eventually hang waiting for database connections
Here's a possible solution - I've thought about it as much as the
possible scenario.
1. Eliminate autoReconnect=true from the URL
2. Make sure connections are closed in the Quartz scheduler job
a. Watch the connections on MySQL
b. Monitor with JMX
3. Fix connection leaks where appropriate
4. Make sure there are no long-running (longer than 60 sec) queries
5. Make sure your database can handle the load
I'm sure there are more things going on, but this is as far as my
crystal ball sees.
. . . . just my two cents.
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
