Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0
Hello Mark, I sent the thread dump directly to you. On 9/8/2018 11:36 AM, Mark Thomas wrote: On 07/09/18 22:56, Bruce Milner wrote: Hello, I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason for not using out-of-the-box is that the existing code relies on changing catalogs at runtime reusing an existing connection. The original design was to use multiple databases using the same connection and this cannot be changed. I recently replaced a lot of hand crafted code with the commons-pool2 implementation. The issue I have is that one server I manage went into a state where there are plenty of connections, but none are being returned to the pool. They are all stuck on a lock inside of GenericKeyedObjectPool.returnObject. The config is basically GenericKeyedObjectPoolConfig config = new GenericKeyedObjectPoolConfig(); config.setBlockWhenExhausted(true); config.setMaxTotal(120); config.setMaxTotalPerKey(60); config.setTestOnBorrow(true); config.setTimeBetweenEvictionRunsMillis(6); config.setMinEvictableIdleTimeMillis(0); // don't starve connections because of catalog switches. /** * For database connections, use FIFO so that we get rid of older connections first before newer ones. */ config.setLifo(false); return new GenericKeyedObjectPool(new PooledConnectionFactory(), config); There are 150 of these threads waiting on a lock to release connections java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0006471cd7d8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389) at org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849) at org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551) at com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358) at com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141) at com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480) and 158 of these threads waiting to open connections. java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0006471cd7d8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560) at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356) at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281) at com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197) at com.ilrn.util.sql.Database.getConnection(Database.java:1273) For the "- parking to wait for <0x0006471cd7d8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155. Though I see no other messages with that object in the thread dump. Has anyone run into this? It seems like some sort of deadlock. Do you still have the full thread dump? Can you post it somewhere (where we can look at it)? Mark - To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional commands, e-mail: user-h...@commons.apache.org -- Bruce Milner Senior Software Developer (Emberex) - To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional
Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0
Hello Bruce, This sounds a bit like a discussion we had about missing wakeups. I think it’s was related to depleted pools. Didn’t find the discussion, hopefully somebody else recalls the conditions? I think it was not fixed. Gruss Bernd -- http://bernd.eckenfels.net Von: Bruce Milner Gesendet: Samstag, September 8, 2018 8:19 PM An: user@commons.apache.org Betreff: Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0 Hello, I did a while back, but my understanding of DBCP is that it has one pool per database and we have thousands. With the number of nodes serving the application multiplied by the number of databases, it could easily exceed maximum number of connections to existing SQL database server. The individual databases are mostly shared by one database server each have individual schemas. We keep track of the database URL and switch the connection via connector.setCatalog(). We also have some that co-exist, so the connection pool has the smarts to decide if it needs a catalog change. The commons pool has been working great so far, and this is the only case I have see where we ended up in this state. We haven't seen this with load tests, but this once in production. I was hoping if this exposed a bug, could get fixed in the pool code. I don't have a reproduction case at this time. I forgot to mention that the environment is java 8 141 with Connector/J 5.1.45 --bruce On 9/7/2018 5:05 PM, Gary Gregory wrote: > Hi, > > A side question: Have you tried Apache Commons DBCP (which is based on > Commons Pool)? > > https://commons.apache.org/proper/commons-dbcp/ > > Gary > > On Fri, Sep 7, 2018 at 5:24 PM Bruce Milner > wrote: > >> Hello, >> >> I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason >> for not using out-of-the-box is that the existing code relies on >> changing catalogs at runtime reusing an existing connection. The >> original design was to use multiple databases using the same connection >> and this cannot be changed. >> >> I recently replaced a lot of hand crafted code with the commons-pool2 >> implementation. >> >> The issue I have is that one server I manage went into a state where >> there are plenty of connections, but none are being returned to the >> pool. They are all stuck on a lock inside of >> GenericKeyedObjectPool.returnObject. >> >> The config is basically >> GenericKeyedObjectPoolConfig config = new >> GenericKeyedObjectPoolConfig(); >> config.setBlockWhenExhausted(true); >> config.setMaxTotal(120); >> config.setMaxTotalPerKey(60); >> config.setTestOnBorrow(true); >> config.setTimeBetweenEvictionRunsMillis(6); >> config.setMinEvictableIdleTimeMillis(0); // don't starve >> connections because of catalog switches. >> /** >> * For database connections, use FIFO so that we get rid of >> older connections first before newer ones. >> */ >> config.setLifo(false); >> return new GenericKeyedObjectPool(new >> PooledConnectionFactory(), config); >> >> There are 150 of these threads waiting on a lock to release connections >> java.lang.Thread.State: WAITING (parking) >> at sun.misc.Unsafe.park(Native Method) >> - parking to wait for <0x0006471cd7d8> (a >> java.util.concurrent.locks.ReentrantLock$NonfairSync) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) >> at >> >> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) >> at >> >> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) >> at >> >> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) >> at >> >> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) >> at >> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) >> at >> >> org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389) >> at >> >> org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849) >> at >> >> org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551) >> at >> >> com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358) >> at >> >> com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141) >> at >> >> com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480) >> >> and 158 of these threads waiting to open connections. >> java.lang.Thread.State: WAITING (parking) >> at sun.misc.Unsafe.park(Native Method) >> - parking to wait for <0x0006471cd7d8> (a >> java.util.concurrent.locks.ReentrantLock$NonfairSync) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) >> at >> >> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) >> at >> >>
Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0
On 07/09/18 22:56, Bruce Milner wrote: > Hello, > > I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason > for not using out-of-the-box is that the existing code relies on > changing catalogs at runtime reusing an existing connection. The > original design was to use multiple databases using the same connection > and this cannot be changed. > > I recently replaced a lot of hand crafted code with the commons-pool2 > implementation. > > The issue I have is that one server I manage went into a state where > there are plenty of connections, but none are being returned to the > pool. They are all stuck on a lock inside of > GenericKeyedObjectPool.returnObject. > > The config is basically > GenericKeyedObjectPoolConfig config = new > GenericKeyedObjectPoolConfig(); > config.setBlockWhenExhausted(true); > config.setMaxTotal(120); > config.setMaxTotalPerKey(60); > config.setTestOnBorrow(true); > config.setTimeBetweenEvictionRunsMillis(6); > config.setMinEvictableIdleTimeMillis(0); // don't starve > connections because of catalog switches. > /** > * For database connections, use FIFO so that we get rid of > older connections first before newer ones. > */ > config.setLifo(false); > return new GenericKeyedObjectPool(new > PooledConnectionFactory(), config); > > There are 150 of these threads waiting on a lock to release connections > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0006471cd7d8> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) > > at > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) > > at > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) > at > org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389) > > at > org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849) > > at > org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551) > > at > com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358) > > at > com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141) > > at > com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480) > > > and 158 of these threads waiting to open connections. > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0006471cd7d8> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) > > at > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) > > at > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) > at > org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560) > > at > org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356) > > at > org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281) > > at > com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197) > > at com.ilrn.util.sql.Database.getConnection(Database.java:1273) > > For the "- parking to wait for <0x0006471cd7d8> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155. > Though I see no other messages with that object in the thread dump. > > Has anyone run into this? It seems like some sort of deadlock. Do you still have the full thread dump? Can you post it somewhere (where we can look at it)? Mark - To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional commands, e-mail: user-h...@commons.apache.org
Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0
Hello, I did a while back, but my understanding of DBCP is that it has one pool per database and we have thousands. With the number of nodes serving the application multiplied by the number of databases, it could easily exceed maximum number of connections to existing SQL database server. The individual databases are mostly shared by one database server each have individual schemas. We keep track of the database URL and switch the connection via connector.setCatalog(). We also have some that co-exist, so the connection pool has the smarts to decide if it needs a catalog change. The commons pool has been working great so far, and this is the only case I have see where we ended up in this state. We haven't seen this with load tests, but this once in production. I was hoping if this exposed a bug, could get fixed in the pool code. I don't have a reproduction case at this time. I forgot to mention that the environment is java 8 141 with Connector/J 5.1.45 --bruce On 9/7/2018 5:05 PM, Gary Gregory wrote: Hi, A side question: Have you tried Apache Commons DBCP (which is based on Commons Pool)? https://commons.apache.org/proper/commons-dbcp/ Gary On Fri, Sep 7, 2018 at 5:24 PM Bruce Milner wrote: Hello, I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason for not using out-of-the-box is that the existing code relies on changing catalogs at runtime reusing an existing connection. The original design was to use multiple databases using the same connection and this cannot be changed. I recently replaced a lot of hand crafted code with the commons-pool2 implementation. The issue I have is that one server I manage went into a state where there are plenty of connections, but none are being returned to the pool. They are all stuck on a lock inside of GenericKeyedObjectPool.returnObject. The config is basically GenericKeyedObjectPoolConfig config = new GenericKeyedObjectPoolConfig(); config.setBlockWhenExhausted(true); config.setMaxTotal(120); config.setMaxTotalPerKey(60); config.setTestOnBorrow(true); config.setTimeBetweenEvictionRunsMillis(6); config.setMinEvictableIdleTimeMillis(0); // don't starve connections because of catalog switches. /** * For database connections, use FIFO so that we get rid of older connections first before newer ones. */ config.setLifo(false); return new GenericKeyedObjectPool(new PooledConnectionFactory(), config); There are 150 of these threads waiting on a lock to release connections java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0006471cd7d8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389) at org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849) at org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551) at com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358) at com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141) at com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480) and 158 of these threads waiting to open connections. java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0006471cd7d8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
