Re: Tomcat becoming almost completely non-responsive
On Tuesday 05 October 2004 6:53 pm, you wrote: > Looks like you're using an old version of the JDBC driver that uses > BufferedInputStreams by default. There is a 'feature' (many call a bug) > in BufferedInputStreams that causes them in some cases to want to read a > full buffer's worth of data when you're only asking for some portion of > it...That's what's happening here. > > I'd try downloading something more recent for a JDBC driver (like > Connector/J 3.0.15) and see where that gets you, to start with. Mark, That's exactly what I needed to know! I didn't consider that version (3.0.11) too old, but hadn't checked your site to see if a newer one was available in the last couple of months. Thank you! --Kaleb - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat becoming almost completely non-responsive
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Kaleb Pederson wrote: > Hello, > > I have an interesting problem. After a while, tomcat (5.0.27) becomes almost > completely non-responsive. If I telnet in to port 8009 (I'm using apache > and mod_jk2), I get no response, at least not within the default timeout. If > a browse to a page, I will generally, after about 4-5 minutes, see a page > returned. > > To narrow down the slowness, I generated a full thread dump, and found the > following information: > > [ see attachment for more info] > Total threads: 180 > executeQuery: 2 // executing a db query > validateConnection: 0 // trying to validate their connection > validateObect: 48 // in commons.dbcp.PoolableConnectionFactory.validateObject > socketAccept: 3 // accepting a socket > socketRead0: 10 // reading a socket > ReferenceQueue: 1 > ThreadPool$MonitorRunnable: 2 > borrowObject and Object.wait: 85 // trying to get an object from the pool > Object.wait: 20 // threads just waiting around > Remaining: 9 // misc. threads > > My database connection is setup so that I have 50 allowed connections, which > matches my 48 in validateObject and 2 executing queries. However, when I > query the database status, I see 2 active threads and the rest are > 'sleeping', just waiting around, as they would be if the connection pool > hadn't released them yet. > > So, why would there be 48 connections that seemed locked and weren't querying > the DB? And then the other 85 that were seemingly waiting on the 45? Any > ideas what might be going on? The DB is ready? I have log abandoned turned > and an haven't seen a problem yet. If the load drops sufficiently on the > server, everything eventually returns back to normal, otherwise it takes 5-10 > minutes to get a response from the server. > > I have attached an abbreviated form of the thread dump which should provide > all the critical information and can provide as much other information as is > necessary. > > Thanks for the help. *All* suggestions welcome ;) > > --Kaleb > > > > > Total threads: 180 > executeQuery: 2 > validateConnection: 0 > validateObect: 48 > socketAccept: 3 > socketRead0: 10 > ReferenceQueue: 1 > ThreadPool$MonitorRunnable: 2 > borrowObject and Object.wait: 85 > borrowObject: 0 > Object.wait: 20 > Remaining: 9 > *** > 2 like "TP-Processor296" daemon prio=1 tid=0x6ea04a90 nid=0x5e87 runnable > [738f6000..738f787c] > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:129) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:183) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:222) > at java.io.BufferedInputStream.read(BufferedInputStream.java:277) [snip] Kaleb, Looks like you're using an old version of the JDBC driver that uses BufferedInputStreams by default. There is a 'feature' (many call a bug) in BufferedInputStreams that causes them in some cases to want to read a full buffer's worth of data when you're only asking for some portion of it...That's what's happening here. I'd try downloading something more recent for a JDBC driver (like Connector/J 3.0.15) and see where that gets you, to start with. -Mark - -- Mr. Mark Matthews MySQL AB, Software Development Manager, J2EE and Windows Platforms Office: +1 708 332 0507 www.mysql.com MySQL Guide to Lower TCO http://www.mysql.com/it-resources/white-papers/tco.php -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.3 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFBY1AztvXNTca6JD8RAvEIAJ4p5Fi9QIwhNTlNslLMW6cKGhmUpgCeP2JJ RurwXfMfDzSEGTRLqssk4b4= =s3u9 -END PGP SIGNATURE- - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat becoming almost completely non-responsive
On Monday 04 October 2004 1:08 pm, Phillip Qin wrote: > Can you add the stuff from my email? I added them to the file and will restart the server when it will cause the least trouble (or next time it goes down). What extra information do you believe this provide that I should be looking for? Thanks again. --Kaleb > -Original Message- > From: Kaleb Pederson [mailto:[EMAIL PROTECTED] > Sent: October 4, 2004 12:40 PM > To: Tomcat Users List > Subject: Re: Tomcat becoming almost completely non-responsive > > > My datasource definition is pretty basic and is as follows: > > > type="javax.sql.DataSource" /> > > > removeAbandoned > true > > > removeAbandonedTimeout > 60 > > > logAbandoned > true > > > factory > org.apache.commons.dbcp.BasicDataSourceFactory > > > username > redirect_user > > > password > redirect_pass > > > driverClassName > com.mysql.jdbc.jdbc2.optional.MysqlDataSource > > > url > jdbc:mysql://myhost/redirects > > > maxActive > 50 > > > maxIdle > 4 > > > validationQuery > show tables > > > > Thanks for the help. > > --Kaleb > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat becoming almost completely non-responsive
Can you add the stuff from my email?
-Original Message-
From: Kaleb Pederson [mailto:[EMAIL PROTECTED]
Sent: October 4, 2004 12:40 PM
To: Tomcat Users List
Subject: Re: Tomcat becoming almost completely non-responsive
My datasource definition is pretty basic and is as follows:
removeAbandoned
true
removeAbandonedTimeout
60
logAbandoned
true
factory
org.apache.commons.dbcp.BasicDataSourceFactory
username
redirect_user
password
redirect_pass
driverClassName
com.mysql.jdbc.jdbc2.optional.MysqlDataSource
url
jdbc:mysql://myhost/redirects
maxActive
50
maxIdle
4
validationQuery
show tables
Thanks for the help.
--Kaleb
On Monday 04 October 2004 8:54 am, Phillip Qin wrote:
> Post your datasource resource definition. You may miss something like
>
>
> testOnBorrow
> true
>
>
> testOnReturn
> true
>
>
> minEvictableIdleTimeMillis
> -1
>
>
> timeBetweenEvictionRunsMillis
> 30
>
>
> numTestsPerEvictionRun
> 1
>
>
> testWhileIdle
> true
>
>
>
> -Original Message-
> From: Kaleb Pederson [mailto:[EMAIL PROTECTED]
> Sent: October 4, 2004 11:19 AM
> To: Tomcat Users List
> Subject: Re: Tomcat becoming almost completely non-responsive
>
>
> John,
>
> Thanks for the feedback. My code is actually quite simple and I have
> everything factored out to single function that handles the DB code.
> I'll paste it below for you, and others to look at. *Please let me
> know* if there is an exit path that I somehow missed.
>
> I already have tomcat setup to log abandoned connections. At your
> instigation, I grep'd the logs and found a few instances of it in the
> logs which I was unaware of. Each one of them corresponds to a time
> we had high load on the server, so I, hopefully not niavely ;), am
> guesssing that the abandoned timeout was shorter (currently 60
> seconds) than the time it took the server to respond to the request
> (up to 5 minutes). I guess I should up
>
> the abandoned timeout as that could flame the problem.
>
> If I'm interpreting the thread dump and mysqladmin processlist
> information correctly, what I'm seeing is that MySQL considers the DB
> connection to be sleeping, and that 48 of my active db threads were in
> object.wait() after a call to AbandonedObjectPool.borrowObject()
> which seems to get called in every case after a call to
> DataSource.getConnection():
>
> // note that I removed the org.apache.commons prefix
> 48 like "TP-Processor312" daemon prio=1 tid=0x70331680 nid=0x5e87
>waiting for monitor entry [6df79000..6df7a87c] at
>
dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.jav
>a
>
> :316)
>
> - waiting to lock <0x7b071438> (a dbcp.PoolableConnectionFactory)
> at
pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:833)
> at dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:119)
> at dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:140)
> at dbcp.BasicDataSource.getConnection(BasicDataSource.java:518)
> // the function in my filter that checks the DB.
> at RequestFilterServlet.getUrlMapping(RequestFilterServlet.java:187)
> ...
>
> Here's the code that handles the db:
>
> protected RedirectInfo getUrlMapping(String requestURI) {
>
> RedirectInfo retval = null;
> Connection conn = null;
> Statement statement = null;
> ResultSet result = null;
>
> try {
> conn = ds.getConnection();
>statement = conn.createStatement();
>statement.setEscapeProcessing(true);
>result = statement.executeQuery(QUERY_UNIQUE_MATCH +
> quote(requestURI));
>
>if (!result.first()) {
> result = statement.executeQuery(QUERY_CHILD_MATCH_1 +
> quote(requestURI)
> +
> QUERY_CHILD_MATCH_2);
>}
>
>// get the information
>if (result.first()) {
> retval = new RedirectInfo();
> retval.setURL(result.getString(1));
> retval.setCode(result.getInt(2));
> debug(requestURI + " -> " + retval.getURL() + " [" +
> Integer.toString(retval.getCode()) + "]");
>} else {
> debug("No matching redirect URL found for " + requestURI);
>}
> } catch(SQLException e) {
>log("SQLException caught: " + e.getMessage(), e);
> } finally {
>try {
> // cleanup after myself
> if (statement != null
Re: Tomcat becoming almost completely non-responsive
Has anybody else had success or failures with the com.mysql.jdbc.jdbc2.optional.MysqlDataSource driver and DBCP? Does the ...DBCP driver correctly integrate with the com.mysql.Driver class so that connection pooling is still valid? What I seem to remember from my initial research in setting it up (which was quite a while ago) was that I needed to use the former, ...optional.MysqlDataSource. Thanks for all the feedback and help. --Kaleb On Monday 04 October 2004 10:34 am, David Smith wrote: > I think I see the problem. Don't use > com.mysql.jdbc.jdbc2.optional.MysqlDataSource in your config. Instead, > use com.mysql.jdbc.Driver. I tried using the same one you did at first > and it never worked with DBCP for me. > > --David > > Kaleb Pederson wrote: > >My datasource definition is pretty basic and is as follows: > > > > > >>type="javax.sql.DataSource" /> > > > > > > removeAbandoned > > true > > > > > > removeAbandonedTimeout > > 60 > > > > > > logAbandoned > > true > > > > > > factory > > org.apache.commons.dbcp.BasicDataSourceFactory > > > > > > username > > redirect_user > > > > > > password > > redirect_pass > > > > > > driverClassName > > com.mysql.jdbc.jdbc2.optional.MysqlDataSource > > > > > > url > > jdbc:mysql://myhost/redirects > > > > > > maxActive > > 50 > > > > > > maxIdle > > 4 > > > > > > validationQuery > > show tables > > > > > > > >Thanks for the help. > > > >--Kaleb > > > >On Monday 04 October 2004 8:54 am, Phillip Qin wrote: > >>Post your datasource resource definition. You may miss something like > >> > >> > >> testOnBorrow > >> true > >> > >> > >> testOnReturn > >> true > >> > >> > >> minEvictableIdleTimeMillis > >> -1 > >> > >> > >> timeBetweenEvictionRunsMillis > >> 30 > >> > >> > >> numTestsPerEvictionRun > >> 1 > >> > >> > >> testWhileIdle > >> true > >> > >> > >> > >>-Original Message- > >>From: Kaleb Pederson [mailto:[EMAIL PROTECTED] > >>Sent: October 4, 2004 11:19 AM > >>To: Tomcat Users List > >>Subject: Re: Tomcat becoming almost completely non-responsive > >> > >> > >>John, > >> > >>Thanks for the feedback. My code is actually quite simple and I have > >>everything factored out to single function that handles the DB code. > >> I'll paste it below for you, and others to look at. *Please let me > >> know* if there > >>is an exit path that I somehow missed. > >> > >>I already have tomcat setup to log abandoned connections. At your > >>instigation, I grep'd the logs and found a few instances of it in the > >> logs which I was unaware of. Each one of them corresponds to a time we > >> had high load on the server, so I, hopefully not niavely ;), am > >> guesssing that the abandoned timeout was shorter (currently 60 seconds) > >> than the time it took the server to respond to the request (up to 5 > >> minutes). I guess I should up > >> > >>the abandoned timeout as that could flame the problem. > >> > >>If I'm interpreting the thread dump and mysqladmin processlist > >> information correctly, what I'm seeing is that MySQL considers the DB > >> connection to be sleeping, and that 48 of my active db threads were in > >> object.wait() after a call to AbandonedObjectPool.borrowObject() which > >> seems to get called in every case after a call to > >> DataSource.getConnection(): > >> > >>// note that I removed the org.apache.commons prefix > >>48 like "TP-Processor312" daemon prio=1 tid=0x70331680 nid=0x5e87 waiting > >>for > >>monitor entry [6df79000..6df7a87c] at > >>dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.j > >>av a > >> > >>:316) > >> > >>- waiting to lock <0x7b071438> (a dbcp.PoolableConnectionFactory) > >>at > >> pool.impl.Generi
Re: Tomcat becoming almost completely non-responsive
I think I see the problem. Don't use
com.mysql.jdbc.jdbc2.optional.MysqlDataSource in your config. Instead,
use com.mysql.jdbc.Driver. I tried using the same one you did at first
and it never worked with DBCP for me.
--David
Kaleb Pederson wrote:
My datasource definition is pretty basic and is as follows:
removeAbandoned
true
removeAbandonedTimeout
60
logAbandoned
true
factory
org.apache.commons.dbcp.BasicDataSourceFactory
username
redirect_user
password
redirect_pass
driverClassName
com.mysql.jdbc.jdbc2.optional.MysqlDataSource
url
jdbc:mysql://myhost/redirects
maxActive
50
maxIdle
4
validationQuery
show tables
Thanks for the help.
--Kaleb
On Monday 04 October 2004 8:54 am, Phillip Qin wrote:
Post your datasource resource definition. You may miss something like
testOnBorrow
true
testOnReturn
true
minEvictableIdleTimeMillis
-1
timeBetweenEvictionRunsMillis
30
numTestsPerEvictionRun
1
testWhileIdle
true
-Original Message-
From: Kaleb Pederson [mailto:[EMAIL PROTECTED]
Sent: October 4, 2004 11:19 AM
To: Tomcat Users List
Subject: Re: Tomcat becoming almost completely non-responsive
John,
Thanks for the feedback. My code is actually quite simple and I have
everything factored out to single function that handles the DB code. I'll
paste it below for you, and others to look at. *Please let me know* if
there
is an exit path that I somehow missed.
I already have tomcat setup to log abandoned connections. At your
instigation, I grep'd the logs and found a few instances of it in the logs
which I was unaware of. Each one of them corresponds to a time we had high
load on the server, so I, hopefully not niavely ;), am guesssing that the
abandoned timeout was shorter (currently 60 seconds) than the time it took
the server to respond to the request (up to 5 minutes). I guess I should
up
the abandoned timeout as that could flame the problem.
If I'm interpreting the thread dump and mysqladmin processlist information
correctly, what I'm seeing is that MySQL considers the DB connection to be
sleeping, and that 48 of my active db threads were in object.wait() after a
call to AbandonedObjectPool.borrowObject() which seems to get called in
every case after a call to DataSource.getConnection():
// note that I removed the org.apache.commons prefix
48 like "TP-Processor312" daemon prio=1 tid=0x70331680 nid=0x5e87 waiting
for
monitor entry [6df79000..6df7a87c] at
dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.jav
a
:316)
- waiting to lock <0x7b071438> (a dbcp.PoolableConnectionFactory)
at pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:833)
at dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:119)
at dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:140)
at dbcp.BasicDataSource.getConnection(BasicDataSource.java:518)
// the function in my filter that checks the DB.
at RequestFilterServlet.getUrlMapping(RequestFilterServlet.java:187)
...
Here's the code that handles the db:
protected RedirectInfo getUrlMapping(String requestURI) {
RedirectInfo retval = null;
Connection conn = null;
Statement statement = null;
ResultSet result = null;
try {
conn = ds.getConnection();
statement = conn.createStatement();
statement.setEscapeProcessing(true);
result = statement.executeQuery(QUERY_UNIQUE_MATCH + quote(requestURI));
if (!result.first()) {
result = statement.executeQuery(QUERY_CHILD_MATCH_1 + quote(requestURI)
+
QUERY_CHILD_MATCH_2);
}
// get the information
if (result.first()) {
retval = new RedirectInfo();
retval.setURL(result.getString(1));
retval.setCode(result.getInt(2));
debug(requestURI + " -> " + retval.getURL() + " [" +
Integer.toString(retval.getCode()) + "]");
} else {
debug("No matching redirect URL found for " + requestURI);
}
} catch(SQLException e) {
log("SQLException caught: " + e.getMessage(), e);
} finally {
try {
// cleanup after myself
if (statement != null) statement.close();
if (conn != null) conn.close();
} catch (SQLException e) {
log("Error cleaning up connection/statement: " + e.getMessage(),e);
}
}
return retval;
}
Thanks for all the help.
--Kaleb
On Sunday 03 October 2004 1:58 pm, [EMAIL PROTECTED] wrote:
The most likely reason for your connections not being re-used is that
your java code isn't explicitly closing connections, statements and
result sets after use. If visiting all your code is too onerous a
task, you could add the following parameters to your data-source
configuration:
removeAbandoned
true
removeAbandonedTimeout
60
The effect of the above is to mark any data-p
Re: Tomcat becoming almost completely non-responsive
My datasource definition is pretty basic and is as follows:
removeAbandoned
true
removeAbandonedTimeout
60
logAbandoned
true
factory
org.apache.commons.dbcp.BasicDataSourceFactory
username
redirect_user
password
redirect_pass
driverClassName
com.mysql.jdbc.jdbc2.optional.MysqlDataSource
url
jdbc:mysql://myhost/redirects
maxActive
50
maxIdle
4
validationQuery
show tables
Thanks for the help.
--Kaleb
On Monday 04 October 2004 8:54 am, Phillip Qin wrote:
> Post your datasource resource definition. You may miss something like
>
>
> testOnBorrow
> true
>
>
> testOnReturn
> true
>
>
> minEvictableIdleTimeMillis
> -1
>
>
> timeBetweenEvictionRunsMillis
> 30
>
>
> numTestsPerEvictionRun
> 1
>
>
> testWhileIdle
> true
>
>
>
> -Original Message-
> From: Kaleb Pederson [mailto:[EMAIL PROTECTED]
> Sent: October 4, 2004 11:19 AM
> To: Tomcat Users List
> Subject: Re: Tomcat becoming almost completely non-responsive
>
>
> John,
>
> Thanks for the feedback. My code is actually quite simple and I have
> everything factored out to single function that handles the DB code. I'll
> paste it below for you, and others to look at. *Please let me know* if
> there
> is an exit path that I somehow missed.
>
> I already have tomcat setup to log abandoned connections. At your
> instigation, I grep'd the logs and found a few instances of it in the logs
> which I was unaware of. Each one of them corresponds to a time we had high
> load on the server, so I, hopefully not niavely ;), am guesssing that the
> abandoned timeout was shorter (currently 60 seconds) than the time it took
> the server to respond to the request (up to 5 minutes). I guess I should
> up
>
> the abandoned timeout as that could flame the problem.
>
> If I'm interpreting the thread dump and mysqladmin processlist information
> correctly, what I'm seeing is that MySQL considers the DB connection to be
> sleeping, and that 48 of my active db threads were in object.wait() after a
> call to AbandonedObjectPool.borrowObject() which seems to get called in
> every case after a call to DataSource.getConnection():
>
> // note that I removed the org.apache.commons prefix
> 48 like "TP-Processor312" daemon prio=1 tid=0x70331680 nid=0x5e87 waiting
> for
> monitor entry [6df79000..6df7a87c] at
> dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.jav
>a
>
> :316)
>
> - waiting to lock <0x7b071438> (a dbcp.PoolableConnectionFactory)
> at pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:833)
> at dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:119)
> at dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:140)
> at dbcp.BasicDataSource.getConnection(BasicDataSource.java:518)
> // the function in my filter that checks the DB.
> at RequestFilterServlet.getUrlMapping(RequestFilterServlet.java:187)
> ...
>
> Here's the code that handles the db:
>
> protected RedirectInfo getUrlMapping(String requestURI) {
>
> RedirectInfo retval = null;
> Connection conn = null;
> Statement statement = null;
> ResultSet result = null;
>
> try {
> conn = ds.getConnection();
>statement = conn.createStatement();
>statement.setEscapeProcessing(true);
>result = statement.executeQuery(QUERY_UNIQUE_MATCH + quote(requestURI));
>
>if (!result.first()) {
> result = statement.executeQuery(QUERY_CHILD_MATCH_1 + quote(requestURI)
> +
> QUERY_CHILD_MATCH_2);
>}
>
>// get the information
>if (result.first()) {
> retval = new RedirectInfo();
> retval.setURL(result.getString(1));
> retval.setCode(result.getInt(2));
> debug(requestURI + " -> " + retval.getURL() + " [" +
> Integer.toString(retval.getCode()) + "]");
>} else {
> debug("No matching redirect URL found for " + requestURI);
>}
> } catch(SQLException e) {
>log("SQLException caught: " + e.getMessage(), e);
> } finally {
>try {
> // cleanup after myself
> if (statement != null) statement.close();
> if (conn != null) conn.close();
>} catch (SQLException e) {
> log("Error cleaning up connection/statement: " + e.getMessage(),e);
>}
> }
>
> return retval;
> }
>
> Thanks for
RE: Tomcat becoming almost completely non-responsive
Post your datasource resource definition. You may miss something like
testOnBorrow
true
testOnReturn
true
minEvictableIdleTimeMillis
-1
timeBetweenEvictionRunsMillis
30
numTestsPerEvictionRun
1
testWhileIdle
true
-Original Message-
From: Kaleb Pederson [mailto:[EMAIL PROTECTED]
Sent: October 4, 2004 11:19 AM
To: Tomcat Users List
Subject: Re: Tomcat becoming almost completely non-responsive
John,
Thanks for the feedback. My code is actually quite simple and I have
everything factored out to single function that handles the DB code. I'll
paste it below for you, and others to look at. *Please let me know* if
there
is an exit path that I somehow missed.
I already have tomcat setup to log abandoned connections. At your
instigation, I grep'd the logs and found a few instances of it in the logs
which I was unaware of. Each one of them corresponds to a time we had high
load on the server, so I, hopefully not niavely ;), am guesssing that the
abandoned timeout was shorter (currently 60 seconds) than the time it took
the server to respond to the request (up to 5 minutes). I guess I should up
the abandoned timeout as that could flame the problem.
If I'm interpreting the thread dump and mysqladmin processlist information
correctly, what I'm seeing is that MySQL considers the DB connection to be
sleeping, and that 48 of my active db threads were in object.wait() after a
call to AbandonedObjectPool.borrowObject() which seems to get called in
every case after a call to DataSource.getConnection():
// note that I removed the org.apache.commons prefix
48 like "TP-Processor312" daemon prio=1 tid=0x70331680 nid=0x5e87 waiting
for
monitor entry [6df79000..6df7a87c] at
dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java
:316)
- waiting to lock <0x7b071438> (a dbcp.PoolableConnectionFactory)
at pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:833)
at dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:119)
at dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:140)
at dbcp.BasicDataSource.getConnection(BasicDataSource.java:518)
// the function in my filter that checks the DB.
at RequestFilterServlet.getUrlMapping(RequestFilterServlet.java:187)
...
Here's the code that handles the db:
protected RedirectInfo getUrlMapping(String requestURI) {
RedirectInfo retval = null;
Connection conn = null;
Statement statement = null;
ResultSet result = null;
try {
conn = ds.getConnection();
statement = conn.createStatement();
statement.setEscapeProcessing(true);
result = statement.executeQuery(QUERY_UNIQUE_MATCH + quote(requestURI));
if (!result.first()) {
result = statement.executeQuery(QUERY_CHILD_MATCH_1 + quote(requestURI)
+
QUERY_CHILD_MATCH_2);
}
// get the information
if (result.first()) {
retval = new RedirectInfo();
retval.setURL(result.getString(1));
retval.setCode(result.getInt(2));
debug(requestURI + " -> " + retval.getURL() + " [" +
Integer.toString(retval.getCode()) + "]");
} else {
debug("No matching redirect URL found for " + requestURI);
}
} catch(SQLException e) {
log("SQLException caught: " + e.getMessage(), e);
} finally {
try {
// cleanup after myself
if (statement != null) statement.close();
if (conn != null) conn.close();
} catch (SQLException e) {
log("Error cleaning up connection/statement: " + e.getMessage(),e);
}
}
return retval;
}
Thanks for all the help.
--Kaleb
On Sunday 03 October 2004 1:58 pm, [EMAIL PROTECTED] wrote:
> The most likely reason for your connections not being re-used is that
> your java code isn't explicitly closing connections, statements and
> result sets after use. If visiting all your code is too onerous a
> task, you could add the following parameters to your data-source
> configuration:
>
>
> removeAbandoned
> true
>
>
> removeAbandonedTimeout
> 60
>
>
> The effect of the above is to mark any data-pool connection as
> re-usable after 60 seconds of inactivity. You can add the following to
> log abandonned connection activity:
>
>
> logAbandoned
> true
>
>
> John Thompson
>
> |-+>
> |
> | | Kaleb Pederson |
> | | <[EMAIL PROTECTED]|
> | | wu.edu> |
>
Re: Tomcat becoming almost completely non-responsive
John,
Thanks for the feedback. My code is actually quite simple and I have
everything factored out to single function that handles the DB code. I'll
paste it below for you, and others to look at. *Please let me know* if there
is an exit path that I somehow missed.
I already have tomcat setup to log abandoned connections. At your
instigation, I grep'd the logs and found a few instances of it in the logs
which I was unaware of. Each one of them corresponds to a time we had high
load on the server, so I, hopefully not niavely ;), am guesssing that the
abandoned timeout was shorter (currently 60 seconds) than the time it took
the server to respond to the request (up to 5 minutes). I guess I should up
the abandoned timeout as that could flame the problem.
If I'm interpreting the thread dump and mysqladmin processlist information
correctly, what I'm seeing is that MySQL considers the DB connection to be
sleeping, and that 48 of my active db threads were in object.wait() after a
call to AbandonedObjectPool.borrowObject() which seems to get called in
every case after a call to DataSource.getConnection():
// note that I removed the org.apache.commons prefix
48 like "TP-Processor312" daemon prio=1 tid=0x70331680 nid=0x5e87 waiting for
monitor entry [6df79000..6df7a87c] at
dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:316)
- waiting to lock <0x7b071438> (a dbcp.PoolableConnectionFactory)
at pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:833)
at dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:119)
at dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:140)
at dbcp.BasicDataSource.getConnection(BasicDataSource.java:518)
// the function in my filter that checks the DB.
at RequestFilterServlet.getUrlMapping(RequestFilterServlet.java:187)
...
Here's the code that handles the db:
protected RedirectInfo getUrlMapping(String requestURI) {
RedirectInfo retval = null;
Connection conn = null;
Statement statement = null;
ResultSet result = null;
try {
conn = ds.getConnection();
statement = conn.createStatement();
statement.setEscapeProcessing(true);
result = statement.executeQuery(QUERY_UNIQUE_MATCH + quote(requestURI));
if (!result.first()) {
result = statement.executeQuery(QUERY_CHILD_MATCH_1 + quote(requestURI) +
QUERY_CHILD_MATCH_2);
}
// get the information
if (result.first()) {
retval = new RedirectInfo();
retval.setURL(result.getString(1));
retval.setCode(result.getInt(2));
debug(requestURI + " -> " + retval.getURL() + " [" +
Integer.toString(retval.getCode()) + "]");
} else {
debug("No matching redirect URL found for " + requestURI);
}
} catch(SQLException e) {
log("SQLException caught: " + e.getMessage(), e);
} finally {
try {
// cleanup after myself
if (statement != null) statement.close();
if (conn != null) conn.close();
} catch (SQLException e) {
log("Error cleaning up connection/statement: " + e.getMessage(),e);
}
}
return retval;
}
Thanks for all the help.
--Kaleb
On Sunday 03 October 2004 1:58 pm, [EMAIL PROTECTED] wrote:
> The most likely reason for your connections not being re-used is that your
> java code isn't explicitly closing connections, statements and result sets
> after use.
> If visiting all your code is too onerous a task, you could add the
> following parameters to your data-source configuration:
>
>
> removeAbandoned
> true
>
>
> removeAbandonedTimeout
> 60
>
>
> The effect of the above is to mark any data-pool connection as re-usable
> after 60 seconds of inactivity.
> You can add the following to log abandonned connection activity:
>
>
> logAbandoned
> true
>
>
> John Thompson
>
> |-+>
> |
> | | Kaleb Pederson |
> | | <[EMAIL PROTECTED]|
> | | wu.edu> |
> | |
> | | 02/10/2004 10:07 |
> | | AM |
> | | Please respond to|
> | | "Tomcat Users|
> | | List"|
> |
> |-+>
> |
> >
> >--|
> >
> | To: Tomcat Users List <[EMAIL PROTECTED]>
> | | cc:
> | |
> | Subject: Tomcat becoming almost completely non-responsive
> ||
> |
> >
> >--|
>
> Hello,
>
> I have an interesting problem. After a while, tomcat (5.0.27) becomes
> almost
> completely non-respon
Re: Tomcat becoming almost completely non-responsive
On Friday 01 October 2004 4:29 pm, Kumar, Sunitha wrote: > What command did you use to generate the thread dump? I'm in Linux, so I used kill -QUIT where was the process ID. --Kaleb > -sunitha > > -Original Message- > From: Kaleb Pederson [mailto:[EMAIL PROTECTED] > Sent: Friday, October 01, 2004 3:08 PM > To: Tomcat Users List > Subject: Tomcat becoming almost completely non-responsive > > > Hello, > > I have an interesting problem. After a while, tomcat (5.0.27) becomes > almost > completely non-responsive. If I telnet in to port 8009 (I'm using > apache > and mod_jk2), I get no response, at least not within the default > timeout. If > a browse to a page, I will generally, after about 4-5 minutes, see a > page > returned. > > To narrow down the slowness, I generated a full thread dump, and found > the > following information: > > [ see attachment for more info] > Total threads: 180 > executeQuery: 2 // executing a db query > validateConnection: 0 // trying to validate their connection > validateObect: 48 // in > commons.dbcp.PoolableConnectionFactory.validateObject > socketAccept: 3 // accepting a socket > socketRead0: 10 // reading a socket > ReferenceQueue: 1 > ThreadPool$MonitorRunnable: 2 > borrowObject and Object.wait: 85 // trying to get an object from the > pool > Object.wait: 20 // threads just waiting around > Remaining: 9 // misc. threads > > My database connection is setup so that I have 50 allowed connections, > which > matches my 48 in validateObject and 2 executing queries. However, when > I > query the database status, I see 2 active threads and the rest are > 'sleeping', just waiting around, as they would be if the connection pool > > hadn't released them yet. > > So, why would there be 48 connections that seemed locked and weren't > querying > the DB? And then the other 85 that were seemingly waiting on the 45? > Any > ideas what might be going on? The DB is ready? I have log abandoned > turned > and an haven't seen a problem yet. If the load drops sufficiently on > the > server, everything eventually returns back to normal, otherwise it takes > 5-10 > minutes to get a response from the server. > > I have attached an abbreviated form of the thread dump which should > provide > all the critical information and can provide as much other information > as is > necessary. > > Thanks for the help. *All* suggestions welcome ;) > > --Kaleb > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat becoming almost completely non-responsive
The most likely reason for your connections not being re-used is that your java code isn't explicitly closing connections, statements and result sets after use. If visiting all your code is too onerous a task, you could add the following parameters to your data-source configuration: removeAbandoned true removeAbandonedTimeout 60 The effect of the above is to mark any data-pool connection as re-usable after 60 seconds of inactivity. You can add the following to log abandonned connection activity: logAbandoned true John Thompson |-+> | | Kaleb Pederson | | | <[EMAIL PROTECTED]| | | wu.edu> | | || | | 02/10/2004 10:07 | | | AM | | | Please respond to| | | "Tomcat Users| | | List"| | || |-+> >--| | | | To: Tomcat Users List <[EMAIL PROTECTED]> | | cc: | | Subject: Tomcat becoming almost completely non-responsive | >--| Hello, I have an interesting problem. After a while, tomcat (5.0.27) becomes almost completely non-responsive. If I telnet in to port 8009 (I'm using apache and mod_jk2), I get no response, at least not within the default timeout. If a browse to a page, I will generally, after about 4-5 minutes, see a page returned. To narrow down the slowness, I generated a full thread dump, and found the following information: [ see attachment for more info] Total threads: 180 executeQuery: 2 // executing a db query validateConnection: 0 // trying to validate their connection validateObect: 48 // in commons.dbcp.PoolableConnectionFactory.validateObject socketAccept: 3 // accepting a socket socketRead0: 10 // reading a socket ReferenceQueue: 1 ThreadPool$MonitorRunnable: 2 borrowObject and Object.wait: 85 // trying to get an object from the pool Object.wait: 20 // threads just waiting around Remaining: 9 // misc. threads My database connection is setup so that I have 50 allowed connections, which matches my 48 in validateObject and 2 executing queries. However, when I query the database status, I see 2 active threads and the rest are 'sleeping', just waiting around, as they would be if the connection pool hadn't released them yet. So, why would there be 48 connections that seemed locked and weren't querying the DB? And then the other 85 that were seemingly waiting on the 45? Any ideas what might be going on? The DB is ready? I have log abandoned turned and an haven't seen a problem yet. If the load drops sufficiently on the server, everything eventually returns back to normal, otherwise it takes 5-10 minutes to get a response from the server. I have attached an abbreviated form of the thread dump which should provide all the critical information and can provide as much other information as is necessary. Thanks for the help. *All* suggestions welcome ;) --Kaleb - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat becoming almost completely non-responsive
Kaleb: What command did you use to generate the thread dump? Thanks, -sunitha -Original Message- From: Kaleb Pederson [mailto:[EMAIL PROTECTED] Sent: Friday, October 01, 2004 3:08 PM To: Tomcat Users List Subject: Tomcat becoming almost completely non-responsive Hello, I have an interesting problem. After a while, tomcat (5.0.27) becomes almost completely non-responsive. If I telnet in to port 8009 (I'm using apache and mod_jk2), I get no response, at least not within the default timeout. If a browse to a page, I will generally, after about 4-5 minutes, see a page returned. To narrow down the slowness, I generated a full thread dump, and found the following information: [ see attachment for more info] Total threads: 180 executeQuery: 2 // executing a db query validateConnection: 0 // trying to validate their connection validateObect: 48 // in commons.dbcp.PoolableConnectionFactory.validateObject socketAccept: 3 // accepting a socket socketRead0: 10 // reading a socket ReferenceQueue: 1 ThreadPool$MonitorRunnable: 2 borrowObject and Object.wait: 85 // trying to get an object from the pool Object.wait: 20 // threads just waiting around Remaining: 9 // misc. threads My database connection is setup so that I have 50 allowed connections, which matches my 48 in validateObject and 2 executing queries. However, when I query the database status, I see 2 active threads and the rest are 'sleeping', just waiting around, as they would be if the connection pool hadn't released them yet. So, why would there be 48 connections that seemed locked and weren't querying the DB? And then the other 85 that were seemingly waiting on the 45? Any ideas what might be going on? The DB is ready? I have log abandoned turned and an haven't seen a problem yet. If the load drops sufficiently on the server, everything eventually returns back to normal, otherwise it takes 5-10 minutes to get a response from the server. I have attached an abbreviated form of the thread dump which should provide all the critical information and can provide as much other information as is necessary. Thanks for the help. *All* suggestions welcome ;) --Kaleb - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
