Hi, We are having two boxes, with 4 brokers each, which are responsible for one queue. Failover works most of the time, but sometimes after a broker shuts down, the one, which is waiting for the lock throws following exception:
java.io.IOException: Failed to start database 'derbydb', see the next exception for details. at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45) at org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.createAdapter(JDBCPersistenceAdapter.java:294) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getAdapter(JDBCPersistenceAdapter.java:249) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.start(JDBCPersistenceAdapter.java:155) at org.apache.activemq.store.journal.JournalPersistenceAdapter.start(JournalPersistenceAdapter.java:226) at org.apache.activemq.broker.BrokerService.createRegionBroker(BrokerService.java:1549) at org.apache.activemq.broker.BrokerService.createBroker(BrokerService.java:1502) at org.apache.activemq.broker.BrokerService.getBroker(BrokerService.java:531) at org.apache.activemq.broker.BrokerService.start(BrokerService.java:440) at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:46) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:128 ) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1257) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:438) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:383) at java.security.AccessController.doPrivileged(Native Method) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:353) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:245) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:169) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:242) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:400) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:736) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:369) at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64) at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52) at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:91) at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:51) at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71) at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54) at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:112) at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74) at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:51) at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:104) at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:51) at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.activemq.console.Main.runTaskClass(Main.java:222) at org.apache.activemq.console.Main.main(Main.java:106) Caused by: SQL Exception: Failed to start database 'derbydb', see the next exception for details. at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.newSQLException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source) at org.apache.derby.jdbc.Driver30.getNewEmbedConnection(Unknown Source) at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source) at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown Source) at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown Source) at org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54) ... 40 more ERROR: java.lang.RuntimeException: Failed to execute start task. Reason: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested ex eption is java.io.IOException: Failed to start database 'derbydb', see the next exception for details. java.lang.RuntimeException: Failed to execute start task. Reason: org.springframework.beans.factory.BeanCreationException: Error creating bean with n me 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.io.IOException: Failed to start database 'derbydb', see the next exception for details. at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:99) at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:51) at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:104) at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:51) at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.activemq.console.Main.runTaskClass(Main.java:222) at org.apache.activemq.console.Main.main(Main.java:106) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' de ined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.io.IOException: Failed to start database 'derb db', see the next exception for details. at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1260) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:438) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:383) at java.security.AccessController.doPrivileged(Native Method) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:353) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:245) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:169) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:242) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:400) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:736) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:369) at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64) at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52) at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:91) at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:51) at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71) at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54) at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:112) at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74) ... 10 more Caused by: java.io.IOException: Failed to start database 'derbydb', see the next exception for details. at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45) at org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.createAdapter(JDBCPersistenceAdapter.java:294) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getAdapter(JDBCPersistenceAdapter.java:249) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.start(JDBCPersistenceAdapter.java:155) at org.apache.activemq.store.journal.JournalPersistenceAdapter.start(JournalPersistenceAdapter.java:226) at org.apache.activemq.broker.BrokerService.createRegionBroker(BrokerService.java:1549) at org.apache.activemq.broker.BrokerService.createBroker(BrokerService.java:1502) at org.apache.activemq.broker.BrokerService.getBroker(BrokerService.java:531) at org.apache.activemq.broker.BrokerService.start(BrokerService.java:440) at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:46) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:128 ) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1257) ... 29 more Caused by: SQL Exception: Failed to start database 'derbydb', see the next exception for details. at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.newSQLException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source) at org.apache.derby.jdbc.Driver30.getNewEmbedConnection(Unknown Source) at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source) at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown Source) at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown Source) at org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54) ... 40 more ERROR: java.lang.Exception: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBr kerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.io.IOException: Failed to sta t database 'derbydb', see the next exception for details. java.lang.Exception: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerSer ice#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.io.IOException: Failed to start data ase 'derbydb', see the next exception for details. at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:100) at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:51) at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:104) at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:51) at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.activemq.console.Main.runTaskClass(Main.java:222) at org.apache.activemq.console.Main.main(Main.java:106) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' de ined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.io.IOException: Failed to start database 'derb db', see the next exception for details. at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1260) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:438) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:383) at java.security.AccessController.doPrivileged(Native Method) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:353) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:245) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:169) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:242) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:400) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:736) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:369) at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64) at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52) at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:91) at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:51) at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71) at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54) at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:112) at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74) ... 10 more Caused by: java.io.IOException: Failed to start database 'derbydb', see the next exception for details. at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45) at org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.createAdapter(JDBCPersistenceAdapter.java:294) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getAdapter(JDBCPersistenceAdapter.java:249) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.start(JDBCPersistenceAdapter.java:155) at org.apache.activemq.store.journal.JournalPersistenceAdapter.start(JournalPersistenceAdapter.java:226) at org.apache.activemq.broker.BrokerService.createRegionBroker(BrokerService.java:1549) at org.apache.activemq.broker.BrokerService.createBroker(BrokerService.java:1502) at org.apache.activemq.broker.BrokerService.getBroker(BrokerService.java:531) at org.apache.activemq.broker.BrokerService.start(BrokerService.java:440) at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:46) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:128 ) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1257) ... 29 more Caused by: SQL Exception: Failed to start database 'derbydb', see the next exception for details. at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.newSQLException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source) at org.apache.derby.jdbc.Driver30.getNewEmbedConnection(Unknown Source) at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source) at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown Source) at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown Source) at org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54) ... 40 more After that, clients can not establish a connection and the newly started broker, which throws the exception is holding the lock. Restarting the original broker, also doesn't get the lock. It worked fine, after sending about 34 000 000 messages over the night, having two failovers (which worked). This is the config of one the brokers: <broker xmlns="http://activemq.org/config/1.0" brokerName="localhost-My-QUEUE" persistent="false" useJmx="true"> <!-- The transport connectors ActiveMQ will listen to --> <transportConnectors> <transportConnector name="tc-localhost-My-QUEUE" uri="tcp://localhost:61616"/> </transportConnectors> <persistenceAdapter> <journaledJDBC dataDirectory="\\192.0.0.2\My-QUEUE"/> </persistenceAdapter> <managementContext> <managementContext connectorPort="2011"/> </managementContext> </broker> We are using a Windows XP SP2 Share with NTFS. thx in advance for any help -- View this message in context: http://www.nabble.com/SQL-Exception-after-failover-using-shared-filesystem-master-slave-tp16323606s2354p16323606.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.