Hello,
I did a while back, but my understanding of DBCP is that it has one pool
per database and we have thousands.
With the number of nodes serving the application multiplied by the
number of databases, it could easily exceed maximum number of
connections to existing SQL database server. The individual databases
are mostly shared by one database server each have individual schemas.
We keep track of the database URL and switch the connection via
connector.setCatalog(). We also have some that co-exist, so the
connection pool has the smarts to decide if it needs a catalog change.
The commons pool has been working great so far, and this is the only
case I have see where we ended up in this state. We haven't seen this
with load tests, but this once in production.
I was hoping if this exposed a bug, could get fixed in the pool code. I
don't have a reproduction case at this time. I forgot to mention that
the environment is java 8 141 with Connector/J 5.1.45
--bruce
On 9/7/2018 5:05 PM, Gary Gregory wrote:
Hi,
A side question: Have you tried Apache Commons DBCP (which is based on
Commons Pool)?
https://commons.apache.org/proper/commons-dbcp/
Gary
On Fri, Sep 7, 2018 at 5:24 PM Bruce Milner <[email protected]>
wrote:
Hello,
I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
for not using out-of-the-box is that the existing code relies on
changing catalogs at runtime reusing an existing connection. The
original design was to use multiple databases using the same connection
and this cannot be changed.
I recently replaced a lot of hand crafted code with the commons-pool2
implementation.
The issue I have is that one server I manage went into a state where
there are plenty of connections, but none are being returned to the
pool. They are all stuck on a lock inside of
GenericKeyedObjectPool.returnObject.
The config is basically
GenericKeyedObjectPoolConfig config = new
GenericKeyedObjectPoolConfig();
config.setBlockWhenExhausted(true);
config.setMaxTotal(120);
config.setMaxTotalPerKey(60);
config.setTestOnBorrow(true);
config.setTimeBetweenEvictionRunsMillis(60000);
config.setMinEvictableIdleTimeMillis(0); // don't starve
connections because of catalog switches.
/**
* For database connections, use FIFO so that we get rid of
older connections first before newer ones.
*/
config.setLifo(false);
return new GenericKeyedObjectPool<String, PooledConnection>(new
PooledConnectionFactory(), config);
There are 150 of these threads waiting on a lock to release connections
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at
org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
at
com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
and 158 of these threads waiting to open connections.
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at
org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
For the "- parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
Though I see no other messages with that object in the thread dump.
Has anyone run into this? It seems like some sort of deadlock.
--
Bruce Milner
Senior Software Developer (Emberex)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
--
Bruce Milner
Senior Software Developer (Emberex)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
