Hi, In a activemq clustered environment (Mysql DB persisted) whenever there is DB connection glitch (DB is not available for a couple of seconds) at least 1 activemq cluster node gets into a weird state, which eventually will cause the termination of the activemq process. This his how i reproduce this behavior:
1. I have an activemq cluster composed of 2 nodes: Node 1 and Node 2. The messages persistence is done using a MYSQL instance all set and good here. If there are no issue on the network the nodes are properly being activated one at a time with the second node taking over when the first one fails. <broker brokerName="node1" xmlns="http://activemq.apache.org/schema/core"> <persistenceAdapter> <jdbcPersistenceAdapter lockKeepAlivePeriod="1000" lockAcquireSleepInterval="2000" dataSource="#mysql-ds"> <locker> <lease-database-locker lockAcquireSleepInterval="2000"/> </locker> <statements> <statements binaryDataType ="LONGBLOB"/> </statements> </jdbcPersistenceAdapter> </persistenceAdapter> <transportConnectors> <transportConnector name="default" uri="tcp://0.0.0.0:61616"/> </transportConnectors> </broker> <bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> 2. Let's say that currently Node1 is the owner of the lock. At this moment if i try to access the Node2 admin console i will not be able to get anything in the browser window. But i'm able to get the console for Node1 3. If i mess with the DB network (to simulate a network glitch) #> service network restart. If i'm lucky enough Node1 will loose the key and Node2 will become master. At this moment if i check again the admin consoles for the 2 nodes I can see that Node2 admin console is available, but Node1 console will be available too. Is jsut that it will present an error on the page (this behavior is different than the usual slave behavior.) 4. If at this point I stop the Node2 activemq service, I would have expected Node1 to take over. unfortunately Node1 will crash at this point with some java.net.BindException: Address already in use errors. You can find the logs for the failing Node1 below. Any idea what is going on here? Thanks 2017-02-03 13:22:58,193 | DEBUG | Scratch dir for the JSP engine is: /tmp/jetty-0.0.0.0-8161-api-_api-any-3039768726853340463.dir/jsp | org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain 2017-02-03 13:22:58,193 | DEBUG | IMPORTANT: Do not modify the generated servlets | org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain 2017-02-03 13:22:58,194 | DEBUG | Defaulting to use topics: true | org.apache.activemq.web.MessageServletSupport | WrapperSimpleAppMain 2017-02-03 13:22:58,197 | INFO | jolokia-agent: Using policy access restrictor classpath:/jolokia-access.xml | /api | WrapperSimpleAppMain 2017-02-03 13:22:58,203 | DEBUG | InitialContextFactory.getInitialContext() | jndi | WrapperSimpleAppMain 2017-02-03 13:22:58,203 | DEBUG | Created initial context delegate for local namespace:org.eclipse.jetty.jndi.local.localContextRoot@d0793bc | jndi | WrapperSimpleAppMain 2017-02-03 13:22:58,203 | DEBUG | Looking up name="comp/env/jmx/runtime" | jndi | WrapperSimpleAppMain 2017-02-03 13:22:58,203 | DEBUG | Trying thread context classloader | jndi | WrapperSimpleAppMain 2017-02-03 13:22:58,203 | DEBUG | supportDeepBinding=false | jndi | WrapperSimpleAppMain 2017-02-03 13:22:58,203 | DEBUG | Made context comp for classloader: WebAppClassLoader=Apache ActiveMQ REST API@2c29a88d | jndi | WrapperSimpleAppMain 2017-02-03 13:22:58,203 | DEBUG | Looking up name="env/jmx/runtime" | jndi | WrapperSimpleAppMain 2017-02-03 13:22:58,207 | INFO | jolokia-agent: jolokia:type=Config is already registered. Adding it with jolokia:type=Config,uuid=a5975fe0-a299-4be1-85c0-7431ef20c86c, but you should revise your setup in order to either use a qualifier or ensure, that only a single agent gets registered (otherwise history functionality might not work) | /api | WrapperSimpleAppMain 2017-02-03 13:22:58,207 | INFO | jolokia-agent: Cannot register (legacy) MBean handler for config store with name jmx4perl:type=Config since it already exists. This is the case if another agent has been already started within the same JVM. The registration is skipped. | /api | WrapperSimpleAppMain 2017-02-03 13:22:58,207 | INFO | jolokia-agent: Jolokia Discovery MBean registration is skipped because there is already one registered. | /api | WrapperSimpleAppMain 2017-02-03 13:22:58,208 | WARN | FAILED ServerConnector@1175f42f{HTTP/1.1}{0.0.0.0:8161}: java.net.BindException: Address already in use | org.eclipse.jetty.util.component.AbstractLifeCycle | WrapperSimpleAppMain java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method)[:1.8.0_102] at sun.nio.ch.Net.bind(Net.java:433)[:1.8.0_102] at sun.nio.ch.Net.bind(Net.java:425)[:1.8.0_102] at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)[:1.8.0_102] at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)[:1.8.0_102] at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.Server.doStart(Server.java:366)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)[spring-core-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.config.MethodInvokingBean.invokeWithTargetException(MethodInvokingBean.java:119)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.config.MethodInvokingFactoryBean.afterPropertiesSet(MethodInvokingFactoryBean.java:106)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1631)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1568)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18] at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18] at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1] at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1] at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1] at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1] at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] at org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.14.1] at org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)[wrapper.jar:3.2.3] at java.lang.Thread.run(Thread.java:745)[:1.8.0_102] 2017-02-03 13:22:58,210 | WARN | FAILED org.eclipse.jetty.server.Server@5f70f898: java.net.BindException: Address already in use | org.eclipse.jetty.util.component.AbstractLifeCycle | WrapperSimpleAppMain java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method)[:1.8.0_102] at sun.nio.ch.Net.bind(Net.java:433)[:1.8.0_102] at sun.nio.ch.Net.bind(Net.java:425)[:1.8.0_102] at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)[:1.8.0_102] at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)[:1.8.0_102] at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.server.Server.doStart(Server.java:366)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)[spring-core-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.config.MethodInvokingBean.invokeWithTargetException(MethodInvokingBean.java:119)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.config.MethodInvokingFactoryBean.afterPropertiesSet(MethodInvokingFactoryBean.java:106)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1631)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1568)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18] at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18] at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1] at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1] at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1] at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1] at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] at org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.14.1] at org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)[wrapper.jar:3.2.3] at java.lang.Thread.run(Thread.java:745)[:1.8.0_102] 2017-02-03 13:22:58,211 | WARN | Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'invokeStart' defined in class path resource [jetty.xml]: Invocation of init method failed; nested exception is java.net.BindException: Address already in use | org.apache.activemq.xbean.XBeanBrokerFactory$1 | WrapperSimpleAppMain 2017-02-03 13:22:58,214 | DEBUG | JspServlet.destroy() | org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain 2017-02-03 13:22:58,217 | INFO | Destroying Spring FrameworkServlet 'dispatcher' | /admin | WrapperSimpleAppMain 2017-02-03 13:22:58,218 | DEBUG | JspServlet.destroy() | org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain 2017-02-03 13:22:58,222 | INFO | Apache ActiveMQ 5.14.1 (jdbcBroker2, ID:mtl-nes-dev-web-48232-1486146033598-0:2) is shutting down | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain 2017-02-03 13:22:58,223 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,connector=clientConnectors,connectorName=default | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain 2017-02-03 13:22:58,225 | TRACE | Execute[ActiveMQ BrokerService[jdbcBroker2] Task] runnable: org.apache.activemq.broker.TransportConnection$4@7af5520a | org.apache.activemq.thread.TaskRunnerFactory | WrapperSimpleAppMain 2017-02-03 13:22:58,225 | TRACE | Created and running thread[ActiveMQ BrokerService[jdbcBroker2] Task-2]: Thread[ActiveMQ BrokerService[jdbcBroker2] Task-2,5,main] | org.apache.activemq.thread.TaskRunnerFactory | WrapperSimpleAppMain 2017-02-03 13:22:58,225 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,connector=clientConnectors,connectorName=default,connectionViewType=clientId,connectionName=mtl-nes-dev-cn3 | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain 2017-02-03 13:22:58,225 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,connector=clientConnectors,connectorName=default,connectionViewType=remoteAddress,connectionName=tcp_//10.3.22.198_41192 | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain 2017-02-03 13:22:58,225 | DEBUG | Stopping connection: tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,227 | DEBUG | Stopping transport tcp:///10.3.22.198:41192@61616 | org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,227 | DEBUG | Initialized TaskRunnerFactory[ActiveMQ Task] using ExecutorService: null | org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,227 | TRACE | Execute[ActiveMQ Task] runnable: org.apache.activemq.transport.tcp.TcpTransport$1@7a152e97 | org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,227 | TRACE | Created and running thread[ActiveMQ Task-1]: Thread[ActiveMQ Task-1,5,main] | org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,227 | TRACE | Closing socket Socket[addr=/10.3.22.198,port=41192,localport=61616] | org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1 2017-02-03 13:22:58,228 | DEBUG | Closed socket Socket[addr=/10.3.22.198,port=41192,localport=61616] | org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1 2017-02-03 13:22:58,228 | DEBUG | Stopped transport: tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,228 | TRACE | Shutdown timeout: 1 task: Transport Connection to: tcp://10.3.22.198:41192 | org.apache.activemq.thread.DedicatedTaskRunner | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,228 | TRACE | Run task done: Transport Connection to: tcp://10.3.22.198:41192 | org.apache.activemq.thread.DedicatedTaskRunner | ActiveMQ Connection Dispatcher: tcp://10.3.22.198:41192 2017-02-03 13:22:58,230 | DEBUG | Cleaning up connection resources: tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,230 | DEBUG | remove connection id: ID:mtl-nes-dev-cn3-38468-1486139024804-1:1 | org.apache.activemq.broker.TransportConnection | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,230 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Queue,destinationName=ComputeNodeQueue-Alin,endpoint=Consumer,clientId=mtl-nes-dev-cn3,consumerId=ID_mtl-nes-dev-cn3-38468-1486139024804-1_1_1_1 | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,230 | DEBUG | jdbcBroker2 removing consumer: ID:mtl-nes-dev-cn3-38468-1486139024804-1:1:1:1 for destination: queue://ComputeNodeQueue-Alin | org.apache.activemq.broker.region.AbstractRegion | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,234 | DEBUG | queue://ComputeNodeQueue-Alin remove sub: QueueSubscription: consumer=ID:mtl-nes-dev-cn3-38468-1486139024804-1:1:1:1, destinations=1, dispatched=0, delivered=0, pending=0, prefetch=1, prefetchExtension=0, lastDeliveredSeqId: -2, dequeues: 0, dispatched: 0, inflight: 0, groups: 0 | org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,235 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616 for broker transport URI: tcp://mtl-nes-dev-web:61616 | org.apache.activemq.broker.TransportConnector | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,235 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616 for broker transport URI: tcp://mtl-nes-dev-web:61616 | org.apache.activemq.broker.TransportConnector | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,236 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.TempQueue_ActiveMQ.Advisory.TempTopic,endpoint=Consumer,clientId=mtl-nes-dev-cn3,consumerId=ID_mtl-nes-dev-cn3-38468-1486139024804-1_1_-1_1 | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,236 | DEBUG | jdbcBroker2 removing consumer: ID:mtl-nes-dev-cn3-38468-1486139024804-1:1:-1:1 for destination: ActiveMQ.Advisory.TempQueue,ActiveMQ.Advisory.TempTopic | org.apache.activemq.broker.region.AbstractRegion | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,237 | TRACE | Running task queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0, pending=0 | org.apache.activemq.thread.DedicatedTaskRunner | Queue:ComputeNodeQueue-Alin 2017-02-03 13:22:58,237 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616 for broker transport URI: tcp://mtl-nes-dev-web:61616 | org.apache.activemq.broker.TransportConnector | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,237 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616 for broker transport URI: tcp://mtl-nes-dev-web:61616 | org.apache.activemq.broker.TransportConnector | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,237 | DEBUG | queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0, pending=0 toPageIn: 0, force:false, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0, enqueueCount: 0, dequeueCount: 0, memUsage:0, maxPageSize:200 | org.apache.activemq.broker.region.Queue | Queue:ComputeNodeQueue-Alin 2017-02-03 13:22:58,238 | DEBUG | Connection Stopped: tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection | ActiveMQ BrokerService[jdbcBroker2] Task-2 2017-02-03 13:22:58,238 | INFO | Connector default stopped | org.apache.activemq.broker.TransportConnector | WrapperSimpleAppMain 2017-02-03 13:22:58,239 | TRACE | Shutdown timeout: 0 task: queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0, pending=0 | org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain 2017-02-03 13:22:58,240 | TRACE | Run task done: queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0, pending=0 | org.apache.activemq.thread.DedicatedTaskRunner | Queue:ComputeNodeQueue-Alin 2017-02-03 13:22:58,241 | TRACE | ComputeNodeQueue-Alin,pendingSize:0 resetBatching. last recovered: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1] | org.apache.activemq.store.jdbc.JDBCMessageStore | WrapperSimpleAppMain 2017-02-03 13:22:58,241 | TRACE | Shutdown timeout: 0 task: Topic: destination=ActiveMQ.Advisory.MasterBroker, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain 2017-02-03 13:22:58,242 | TRACE | Shutdown timeout: 0 task: Topic: destination=ActiveMQ.Advisory.Connection, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain 2017-02-03 13:22:58,242 | TRACE | Run task done: Topic: destination=ActiveMQ.Advisory.MasterBroker, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | Topic ActiveMQ.Advisory.MasterBroker 2017-02-03 13:22:58,242 | TRACE | Run task done: Topic: destination=ActiveMQ.Advisory.Connection, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | Topic ActiveMQ.Advisory.Connection 2017-02-03 13:22:58,242 | TRACE | Shutdown timeout: 0 task: Topic: destination=ActiveMQ.Advisory.Queue, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain 2017-02-03 13:22:58,243 | TRACE | Run task done: Topic: destination=ActiveMQ.Advisory.Queue, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | Topic ActiveMQ.Advisory.Queue 2017-02-03 13:22:58,243 | TRACE | Shutdown timeout: 0 task: Topic: destination=ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain 2017-02-03 13:22:58,243 | TRACE | Run task done: Topic: destination=ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin, subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | Topic ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.MasterBroker | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.Connection | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Queue,destinationName=ComputeNodeQueue-Alin | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.Queue | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain 2017-02-03 13:22:58,243 | INFO | PListStore:[/usr/local/Nuance/activemq/bin/linux-x86-64/activemq-data/jdbcBroker2/tmp_storage] stopped | org.apache.activemq.store.kahadb.plist.PListStoreImpl | WrapperSimpleAppMain 2017-02-03 13:22:58,287 | INFO | jdbcBroker2, released lease | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | WrapperSimpleAppMain 2017-02-03 13:22:58,287 | TRACE | Shutdown of ExecutorService: java.util.concurrent.ScheduledThreadPoolExecutor@21e82532[Running, pool size = 4, active threads = 0, queued tasks = 2, completed tasks = 2] with await termination: 0 millis | org.apache.activemq.util.ThreadPoolUtils | WrapperSimpleAppMain 2017-02-03 13:22:58,288 | DEBUG | Shutdown of ExecutorService: java.util.concurrent.ScheduledThreadPoolExecutor@21e82532[Shutting down, pool size = 4, active threads = 0, queued tasks = 0, completed tasks = 2] is shutdown: true and terminated: false took: 0.001 seconds. | org.apache.activemq.util.ThreadPoolUtils | WrapperSimpleAppMain 2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,service=Health | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain 2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2,service=Log4JConfiguration | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain 2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean org.apache.activemq:type=Broker,brokerName=jdbcBroker2 | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain 2017-02-03 13:22:58,288 | DEBUG | Stopping jmx connector | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain 2017-02-03 13:22:58,288 | DEBUG | InitialContextFactory.getInitialContext() | jndi | WrapperSimpleAppMain 2017-02-03 13:22:58,288 | DEBUG | Created initial context delegate for local namespace:org.eclipse.jetty.jndi.local.localContextRoot@574009c8 | jndi | WrapperSimpleAppMain 2017-02-03 13:22:58,292 | DEBUG | Unexported JMX RMI Registry | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain 2017-02-03 13:22:58,292 | DEBUG | Forcing shutdown of ExecutorService: java.util.concurrent.ThreadPoolExecutor@68f23545[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] | org.apache.activemq.util.ThreadPoolUtils | WrapperSimpleAppMain 2017-02-03 13:22:58,292 | TRACE | Shutdown of ExecutorService: java.util.concurrent.ThreadPoolExecutor@68f23545[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] is shutdown: true and terminated: true. | org.apache.activemq.util.ThreadPoolUtils | WrapperSimpleAppMain 2017-02-03 13:22:58,292 | INFO | Apache ActiveMQ 5.14.1 (jdbcBroker2, ID:mtl-nes-dev-web-48232-1486146033598-0:2) uptime 1 minute | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain 2017-02-03 13:22:58,292 | INFO | Apache ActiveMQ 5.14.1 (jdbcBroker2, ID:mtl-nes-dev-web-48232-1486146033598-0:2) is shutdown | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain 2017-02-03 13:22:58,295 | ERROR | Failed to load: class path resource [activemq.xml], reason: Error creating bean with name 'invokeStart' defined in class path resource [jetty.xml]: Invocation of init method failed; nested exception is java.net.BindException: Address already in use | org.apache.activemq.xbean.XBeanBrokerFactory | WrapperSimpleAppMain org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'invokeStart' defined in class path resource [jetty.xml]: Invocation of init method failed; nested exception is java.net.BindException: Address already in use at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1572)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE] at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18] at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18] at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1] at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1] at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1] at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1] at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1] at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] at org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.14.1] at org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_102] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)[wrapper.jar:3.2.3] at java.lang.Thread.run(Thread.java:745)[:1.8.0_102] -- View this message in context: http://activemq.2283324.n4.nabble.com/Failover-problems-when-DB-replication-is-being-used-tp4721586.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.