Hi ,
I think I found a bug ? I will describe my testscase. I'm using sequoia 2.10.8. I have a VirtualDatabase with two backends. My loadbalancer is a RAIDb1_WRR load balancer. During an heavy database process, I cut the network with one of my databases. The databases is disabled è OK. But my process hang è L I checked the threads call stacks and I can see all the VirtualDatabaseWorkerThread were blocked. Below is the stack: INFO | jvm 1 | 2007/07/06 15:32:42 | java.lang.Thread.State: BLOCKED (on object monitor) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.backend.DatabaseBackend.isReadEnabled(DatabaseBackend.java:1497) INFO | jvm 1 | 2007/07/06 15:32:42 | - waiting to lock <0x082fdba8> (a org.continuent.sequoia.controller.backend.DatabaseBackend) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.loadbalancer.raidb1.RAIDb1_WRR.executeWRR(RAIDb1_WRR.java:195) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.loadbalancer.raidb1.RAIDb1_WRR.execSingleBackendReadRequest(RAIDb1_WRR.java:112) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.loadbalancer.raidb1.RAIDb1.statementExecuteQuery(RAIDb1.java:160) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.requestmanager.RequestManager.statementExecuteQuery(RequestManager.java:607) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabase.statementExecuteQuery(VirtualDatabase.java:549) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.statementExecuteQuery(VirtualDatabaseWorkerThread.java:2175) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.run(VirtualDatabaseWorkerThread.java:442) Furthermore, I find another interesting stack: INFO | jvm 1 | 2007/07/06 15:32:42 | java.lang.Thread.State: RUNNABLE INFO | jvm 1 | 2007/07/06 15:32:42 | at java.net.SocketInputStream.socketRead0(Native Method) INFO | jvm 1 | 2007/07/06 15:32:42 | at java.net.SocketInputStream.read(SocketInputStream.java:129) INFO | jvm 1 | 2007/07/06 15:32:42 | at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) INFO | jvm 1 | 2007/07/06 15:32:42 | at java.io.BufferedInputStream.read(BufferedInputStream.java:237) INFO | jvm 1 | 2007/07/06 15:32:42 | - locked <0x08b0fec0> (a java.io.BufferedInputStream) INFO | jvm 1 | 2007/07/06 15:32:42 | at java.io.DataInputStream.readInt(DataInputStream.java:370) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.common.protocol.TypeTag.<init>(TypeTag.java:169) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.driver.Connection.receiveBooleanOrException(Connection.java:2176) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.driver.Connection.receiveSQLWarnings(Connection.java:2112) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.driver.Connection.statementExecuteQuery(Connection.java:2763) INFO | jvm 1 | 2007/07/06 15:32:42 | - locked <0x08afd4f0> (a org.continuent.sequoia.driver.Connection) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.driver.Statement.executeQuery(Statement.java:522) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.driver.PreparedStatement.executeQuery(PreparedStatement.java:168) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.apache.ojb.broker.accesslayer.JdbcAccessImpl.executeQuery(JdbcAccessImpl.java:291) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.apache.ojb.broker.accesslayer.RsQueryObject.performQuery(RsQueryObject.java:72) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.apache.ojb.broker.accesslayer.RsIterator.<init>(RsIterator.java:187) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.apache.ojb.broker.core.RsIteratorFactoryImpl.createRsIterator(RsIteratorFactoryImpl.java:58) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.apache.ojb.broker.core.PersistenceBrokerImpl.getRsIteratorFromQuery(PersistenceBrokerImpl.java:2109) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.apache.ojb.broker.core.PersistenceBrokerImpl.getIteratorFromQuery(PersistenceBrokerImpl.java:1664) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.apache.ojb.broker.core.PersistenceBrokerImpl.getObjectByQuery(PersistenceBrokerImpl.java:1790) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.apache.ojb.broker.core.DelegatingPersistenceBroker.getObjectByQuery(DelegatingPersistenceBroker.java:352) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.apache.ojb.broker.core.DelegatingPersistenceBroker.getObjectByQuery(DelegatingPersistenceBroker.java:352) INFO | jvm 1 | 2007/07/06 15:32:42 | at com.infovista.mserver.DbManager.getValidatedSlave(DbManager.java:1862) INFO | jvm 1 | 2007/07/06 15:32:42 | at com.infovista.mserver.JMXManager.backupDisabled(JMXManager.java:1388) INFO | jvm 1 | 2007/07/06 15:32:42 | at com.infovista.mserver.actions.CJDBCLaunch$2.handleNotification(CJDBCLaunch.java:476) INFO | jvm 1 | 2007/07/06 15:32:42 | at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1732) INFO | jvm 1 | 2007/07/06 15:32:42 | at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257) INFO | jvm 1 | 2007/07/06 15:32:42 | at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322) INFO | jvm 1 | 2007/07/06 15:32:42 | at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307) INFO | jvm 1 | 2007/07/06 15:32:42 | at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.backend.DatabaseBackend.sendNotification(DatabaseBackend.java:1643) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.backend.DatabaseBackend.notifyJMXStateChanged(DatabaseBackend.java:1578) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.backend.DatabaseBackend.setState(DatabaseBackend.java:1736) INFO | jvm 1 | 2007/07/06 15:32:42 | - locked <0x082fdba8> (a org.continuent.sequoia.controller.backend.DatabaseBackend) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.backend.DatabaseBackend.disable(DatabaseBackend.java:1260) INFO | jvm 1 | 2007/07/06 15:32:42 | - locked <0x082fdba8> (a org.continuent.sequoia.controller.backend.DatabaseBackend) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.loadbalancer.raidb1.RAIDb1.disableBackend(RAIDb1.java:1992) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.loadbalancer.tasks.StatementExecuteUpdateTask.executeInTransaction(StatementExecuteUpdateTask.java:295) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.loadbalancer.tasks.StatementExecuteUpdateTask.executeTask(StatementExecuteUpdateTask.java:118) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.loadbalancer.tasks.AbstractTask.execute(AbstractTask.java:140) INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.loadbalancer.BackendWorkerThread.run(BackendWorkerThread.java:197) This last stack shows that Sequoia is trying to disable the fallen database. My application is listening on the state of the backend. When My application is informed, thanks to a JMX notification, that the database is down, it tries to make a db request. Unfortunately, this request will never finished because all the DatabaseWorker threads are blocked in INFO | jvm 1 | 2007/07/06 15:32:42 | at org.continuent.sequoia.controller.backend.DatabaseBackend.isReadEnabled(DatabaseBackend.java:1497) INFO | jvm 1 | 2007/07/06 15:32:42 | - waiting to lock <0x082fdba8> (a org.continuent.sequoia.controller.backend.DatabaseBackend) Last point, the configuration of my virtual database is poolThreads="false" So my question is: Is there a way to avoid this type of issue ? My understanding is that the RAIDb1.disableBackend should call, at the beginning, backend.setState (BackendState.DISABLING) and not db.disable(). The method db.disable() should only be called at the end, when the database is really disabled. Thanks, Xavier.
_______________________________________________ Sequoia mailing list [email protected] https://forge.continuent.org/mailman/listinfo/sequoia
