A follow up on this issue:
We replaced the SimpleDataSource with Apache Commons DBCP but see the same
issue: The pool is getting in some kind of deadlock state where threads are
blocked while trying to get a lock on the pool in order to return or borrow a
connection to or from the pool. Please see the thread dump below.
I'd assume that both borrowObject() and returnObject() always complete promptly
and therefore such a deadlock never occurs, and that in case the database can't
keep up the pool rather gets exhausted, but the behaviour seen here I really
don't understand.
It happens usually after a couple of days at peak times where also the database
is really loaded.
These are our current iBatis settings:
<settings
cacheModelsEnabled="true"
enhancementEnabled="true"
maxRequests="320"
maxSessions="128"
maxTransactions="64"/>
And the DBCP settings:
maxActive=64
maxIdle=16
maxWait=1000
validationQuery=select * from dual
"TP-Processor483" daemon prio=10 tid=0x01fe5000 nid=0x11f7 waiting for monitor
entry [0x8d87e000..0x8d881788]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.commons.pool.impl.GenericObjectPool.returnObject(GenericObjectPool.java:916)
- waiting to lock <0x9ddf7460> (a
org.apache.commons.pool.impl.GenericObjectPool)
at
org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:87)
- locked <0xf83f9eb8> (a org.apache.commons.dbcp.PoolableConnection)
at
org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:181)
at
com.ibatis.sqlmap.engine.transaction.jdbc.JdbcTransaction.close(JdbcTransaction.java:81)
at
com.ibatis.sqlmap.engine.transaction.TransactionManager.end(TransactionManager.java:110)
"TP-Processor382" daemon prio=10 tid=0x026f6000 nid=0x25a waiting for monitor
entry [0x7de7f000..0x7de81808]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:781)
- waiting to lock <0x9ddf7460> (a
org.apache.commons.pool.impl.GenericObjectPool)
at
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
at
com.ibatis.sqlmap.engine.transaction.jdbc.JdbcTransaction.init(JdbcTransaction.java:48)
at
com.ibatis.sqlmap.engine.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:89)
Torsten
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Sent: 25. september 2007 22:50
To: [email protected]
Subject: RE: iBatis settings and datasource configuration
The app is quite critical, it's our customer activation system. It is pretty
serious, and the load is considerable.
We are already considering to go for Apache commons DBCP instead, since we have
only good experience with it in a number of other apps. Your reply supports
this - thanks for that! :-)
Since we have a new release very soon, we'll go with the adjusted settings for
now, but I guess from next release we'll use DBCP. After all, the name
SimpleDataSource kind of suggests that it may not be suitable for larger apps...
Thanks for your helpful advice!
Torsten
From: Adam Zimowski [mailto:[EMAIL PROTECTED]
Sent: 25. september 2007 21:27
How critical is your app? What's the peak load? If this is a serious enterprise
app, I'd consider using alternate data source:
* From Developer Guide (I'm sure you've seen that) *
Note!: SimpleDataSource is quite convenient, efficient and effective.
However, for large enterprise or mission critical applications, it is
recommended that you use an enterprise level DataSource implementation (such as
those that come with App Servers and commercial O/R mapping tools).
Reducing Pool.MaximumCheckoutTime may help to narrow down the issue, as checked
out connections will be eligible for pool sooner than you may have specified
currently. Also, advising your DAOs with something like Spring AOP or AspectJ
to log meaningful usage statistics may help to provide some verbosity to how
your DAO layer is utilized by your web app, especially in regards to the load
it may be experiencing at different times.
In my opinion, the most straightforward way to attack the problem is to try a
different data source implementation. For many years I've been very successfull
with Apache commons DBCP, and given the warning about SimpleDataSource usage in
iBatis documentation I'd definitely try alternate implementation before
investing too much time and effort on what may be a dead-end.
In regards to settings, your suggestion to try higher maxRequests is what I
would also try. I'm not sure if reducing max transactions will make much
difference, because your db server isn't likely to experiencing half of that.
You have max transactions at 64 but it probably never even reaches that limit.
One last thing - although it may be an overkill - is to enable iBatis debugging
via in your logger, which may provide some interesting hints. That however, may
come at cost of significant slowdown to your app (depending on the load which I
don't know what it is).
Regards,
-adam
On 9/24/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> Our J2SE webapp using iBatis with SimpleDataSource running in Apache Tomcat
> occasionally starts to hang. It hits the roof of max. Nr. of threads and
> looking at the thread dump, there are many threads that seem blocked trying
> to get and return(!) connections from and to the pool:
>
> "TP-Processor406" daemon prio=10 tid=0x005fbc00 nid=0xda4 waiting for monitor
> entry [0x7ce7f000..0x7ce81788]
> java.lang.Thread.State: BLOCKED (on object monitor) at
> com.ibatis.common.jdbc.SimpleDataSource.pushConnection(SimpleDataSource.java:521)
> - waiting to lock <0x9e9d37f0> (a java.lang.Object) at
> com.ibatis.common.jdbc.SimpleDataSource.access$100(SimpleDataSource.ja
> va:52) at
> com.ibatis.common.jdbc.SimpleDataSource$SimplePooledConnection.invoke(
> SimpleDataSource.java:954)
> at $Proxy21.close(Unknown Source)
> [...]
>
> It almost looks to me as if SimpleDataSource blocks threads from returning
> connections to the pool, while other threads are trying to get connections
> from it - some kind of deadlock scenario?
>
> Assuming that my observation is wrong, I wonder if our configuration is good.
> We currently have:
>
> iBatis settings
>
> maxRequests="128"
> maxSessions="64"
> maxTransactions="64"
>
> and datasource settings
>
> maxActive=50
> maxIdle=15
>
> Isn't maxTransactions too high? Besides the rules pointed out in the
> documentation, are there rules on how these settings should relate to the
> datasource settings? Something like maxActive should be at least maxSessions
> or something like that?
>
> Would these settings be any good?
>
> maxRequests="320"
> maxSessions="64"
> maxTransactions="32"
>
> Torsten
>
