Please submit a bug in JIRA. Please attach config files for both ActiveMQ and MySQL, to make it easier for someone to reproduce; the database persistence store code doesn't get a lot of love, so it's more likely someone will investigate if you can provide everything needed to reproduce the problem. Does data need to be running for the problem to occur, or can you make it happen without ever connecting a client?
Also, please take thread dumps of broker 1 at each step, and attach them to the JIRA. Hopefully they will shed some light on what is going on. Tim On Feb 3, 2017 12:12 PM, "All in" <ghe...@gmail.com> wrote: > Hi, > In a activemq clustered environment (Mysql DB persisted) whenever there is > DB connection glitch (DB is not available for a couple of seconds) at least > 1 activemq cluster node gets into a weird state, which eventually will > cause the termination of the activemq process. This his how i reproduce > this > behavior: > > 1. I have an activemq cluster composed of 2 nodes: Node 1 and Node 2. The > messages persistence is done using a MYSQL instance all set and good here. > If there are no issue on the network the nodes are properly being activated > one at a time with the second node taking over when the first one fails. > > <broker brokerName="node1" > xmlns="http://activemq.apache.org/schema/core";> > <persistenceAdapter> > <jdbcPersistenceAdapter lockKeepAlivePeriod="1000" > lockAcquireSleepInterval="2000" dataSource="#mysql-ds"> > <locker> > <lease-database-locker lockAcquireSleepInterval=" > 2000"/> > </locker> > <statements> > <statements binaryDataType ="LONGBLOB"/> > </statements> > </jdbcPersistenceAdapter> > </persistenceAdapter> > <transportConnectors> > <transportConnector name="default" uri="tcp://0.0.0.0:61616"/> > </transportConnectors> > </broker> > > <bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource" > destroy-method="close"> > <property name="driverClassName" value="com.mysql.jdbc.Driver"/> > > 2. Let's say that currently Node1 is the owner of the lock. At this moment > if i try to access the Node2 admin console i will not be able to get > anything in the browser window. But i'm able to get the console for Node1 > > 3. If i mess with the DB network (to simulate a network glitch) #> service > network restart. If i'm lucky enough Node1 will loose the key and Node2 > will > become master. At this moment if i check again the admin consoles for the 2 > nodes I can see that Node2 admin console is available, but Node1 console > will be available too. Is jsut that it will present an error on the page > (this behavior is different than the usual slave behavior.) > > 4. If at this point I stop the Node2 activemq service, I would have > expected > Node1 to take over. unfortunately Node1 will crash at this point with some > java.net.BindException: Address already in use errors. You can find the > logs > for the failing Node1 below. > > Any idea what is going on here? > > Thanks > > > 2017-02-03 13:22:58,193 | DEBUG | Scratch dir for the JSP engine is: > /tmp/jetty-0.0.0.0-8161-api-_api-any-3039768726853340463.dir/jsp | > org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain > 2017-02-03 13:22:58,193 | DEBUG | IMPORTANT: Do not modify the generated > servlets | org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain > 2017-02-03 13:22:58,194 | DEBUG | Defaulting to use topics: true | > org.apache.activemq.web.MessageServletSupport | WrapperSimpleAppMain > 2017-02-03 13:22:58,197 | INFO | jolokia-agent: Using policy access > restrictor classpath:/jolokia-access.xml | /api | WrapperSimpleAppMain > 2017-02-03 13:22:58,203 | DEBUG | InitialContextFactory. > getInitialContext() > | jndi | WrapperSimpleAppMain > 2017-02-03 13:22:58,203 | DEBUG | Created initial context delegate for > local > namespace:org.eclipse.jetty.jndi.local.localContextRoot@d0793bc | jndi | > WrapperSimpleAppMain > 2017-02-03 13:22:58,203 | DEBUG | Looking up name="comp/env/jmx/runtime" | > jndi | WrapperSimpleAppMain > 2017-02-03 13:22:58,203 | DEBUG | Trying thread context classloader | jndi > | > WrapperSimpleAppMain > 2017-02-03 13:22:58,203 | DEBUG | supportDeepBinding=false | jndi | > WrapperSimpleAppMain > 2017-02-03 13:22:58,203 | DEBUG | Made context comp for classloader: > WebAppClassLoader=Apache ActiveMQ REST API@2c29a88d | jndi | > WrapperSimpleAppMain > 2017-02-03 13:22:58,203 | DEBUG | Looking up name="env/jmx/runtime" | jndi > | > WrapperSimpleAppMain > 2017-02-03 13:22:58,207 | INFO | jolokia-agent: jolokia:type=Config is > already registered. Adding it with > jolokia:type=Config,uuid=a5975fe0-a299-4be1-85c0-7431ef20c86c, but you > should revise your setup in order to either use a qualifier or ensure, that > only a single agent gets registered (otherwise history functionality might > not work) | /api | WrapperSimpleAppMain > 2017-02-03 13:22:58,207 | INFO | jolokia-agent: Cannot register (legacy) > MBean handler for config store with name jmx4perl:type=Config since it > already exists. This is the case if another agent has been already started > within the same JVM. The registration is skipped. | /api | > WrapperSimpleAppMain > 2017-02-03 13:22:58,207 | INFO | jolokia-agent: Jolokia Discovery MBean > registration is skipped because there is already one registered. | /api | > WrapperSimpleAppMain > 2017-02-03 13:22:58,208 | WARN | FAILED > ServerConnector@1175f42f{HTTP/1.1}{0.0.0.0:8161}: java.net.BindException: > Address already in use | org.eclipse.jetty.util. > component.AbstractLifeCycle > | WrapperSimpleAppMain > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method)[:1.8.0_102] > at sun.nio.ch.Net.bind(Net.java:433)[:1.8.0_102] > at sun.nio.ch.Net.bind(Net.java:425)[:1.8.0_102] > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java: > 223)[:1.8.0_102] > at > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)[: > 1.8.0_102] > at > org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)[ > jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] > at > org.eclipse.jetty.server.AbstractNetworkConnector.doStart( > AbstractNetworkConnector.java:80)[jetty-all-9.2.13.v20150730.jar:9.2.13. > v20150730] > at > org.eclipse.jetty.server.ServerConnector.doStart( > ServerConnector.java:236)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] > at > org.eclipse.jetty.util.component.AbstractLifeCycle. > start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13. > v20150730] > at > org.eclipse.jetty.server.Server.doStart(Server.java:366)[jetty-all-9.2.13. > v20150730.jar:9.2.13.v20150730] > at > org.eclipse.jetty.util.component.AbstractLifeCycle. > start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13. > v20150730] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method)[:1.8.0_102] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62)[:1.8.0_102] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] > at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] > at > org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)[ > spring-core-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.config.MethodInvokingBean. > invokeWithTargetException(MethodInvokingBean.java:119)[ > spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.config.MethodInvokingFactoryBean. > afterPropertiesSet(MethodInvokingFactoryBean.java:106)[spring-beans-4.1.9. > RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.invokeInitMethods(AbstractAutowireCapableBeanFac > tory.java:1631)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.initializeBean(AbstractAutowireCapableBeanFac > tory.java:1568)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.doCreateBean(AbstractAutowireCapableBeanFac > tory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.createBean(AbstractAutowireCapableBeanFac > tory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractBeanFactory$1. > getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9. > RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry. > getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9. > RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean( > AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE. > jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractBeanFactory.getBean( > AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE. > jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.DefaultListableBeanFactory. > preInstantiateSingletons(DefaultListableBeanFactory. > java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.context.support.AbstractApplicationContext. > finishBeanFactoryInitialization(AbstractApplicationContext. > java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.context.support.AbstractApplicationContext.refresh( > AbstractApplicationContext.java:480)[spring-context-4.1. > 9.RELEASE.jar:4.1.9.RELEASE] > at > org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>( > ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18] > at > org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>( > ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18] > at > org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>( > XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1] > at > org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext( > XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1] > at > org.apache.activemq.xbean.XBeanBrokerFactory.createBroker( > XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1] > at > org.apache.activemq.broker.BrokerFactory.createBroker( > BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1] > at > org.apache.activemq.broker.BrokerFactory.createBroker( > BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.StartCommand.runTask( > StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.AbstractCommand. > execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.ShellCommand.runTask( > ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.AbstractCommand. > execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.ShellCommand.main( > ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method)[:1.8.0_102] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62)[:1.8.0_102] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] > at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] > at > org.apache.activemq.console.Main.runTaskClass(Main.java: > 262)[activemq.jar:5.14.1] > at > org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method)[:1.8.0_102] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62)[:1.8.0_102] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] > at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] > at > org.tanukisoftware.wrapper.WrapperSimpleApp.run( > WrapperSimpleApp.java:240)[wrapper.jar:3.2.3] > at java.lang.Thread.run(Thread.java:745)[:1.8.0_102] > 2017-02-03 13:22:58,210 | WARN | FAILED > org.eclipse.jetty.server.Server@5f70f898: java.net.BindException: Address > already in use | org.eclipse.jetty.util.component.AbstractLifeCycle | > WrapperSimpleAppMain > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method)[:1.8.0_102] > at sun.nio.ch.Net.bind(Net.java:433)[:1.8.0_102] > at sun.nio.ch.Net.bind(Net.java:425)[:1.8.0_102] > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java: > 223)[:1.8.0_102] > at > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)[: > 1.8.0_102] > at > org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)[ > jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] > at > org.eclipse.jetty.server.AbstractNetworkConnector.doStart( > AbstractNetworkConnector.java:80)[jetty-all-9.2.13.v20150730.jar:9.2.13. > v20150730] > at > org.eclipse.jetty.server.ServerConnector.doStart( > ServerConnector.java:236)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730] > at > org.eclipse.jetty.util.component.AbstractLifeCycle. > start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13. > v20150730] > at > org.eclipse.jetty.server.Server.doStart(Server.java:366)[jetty-all-9.2.13. > v20150730.jar:9.2.13.v20150730] > at > org.eclipse.jetty.util.component.AbstractLifeCycle. > start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13. > v20150730] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method)[:1.8.0_102] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62)[:1.8.0_102] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] > at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] > at > org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)[ > spring-core-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.config.MethodInvokingBean. > invokeWithTargetException(MethodInvokingBean.java:119)[ > spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.config.MethodInvokingFactoryBean. > afterPropertiesSet(MethodInvokingFactoryBean.java:106)[spring-beans-4.1.9. > RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.invokeInitMethods(AbstractAutowireCapableBeanFac > tory.java:1631)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.initializeBean(AbstractAutowireCapableBeanFac > tory.java:1568)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.doCreateBean(AbstractAutowireCapableBeanFac > tory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.createBean(AbstractAutowireCapableBeanFac > tory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractBeanFactory$1. > getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9. > RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry. > getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9. > RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean( > AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE. > jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractBeanFactory.getBean( > AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE. > jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.DefaultListableBeanFactory. > preInstantiateSingletons(DefaultListableBeanFactory. > java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.context.support.AbstractApplicationContext. > finishBeanFactoryInitialization(AbstractApplicationContext. > java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.context.support.AbstractApplicationContext.refresh( > AbstractApplicationContext.java:480)[spring-context-4.1. > 9.RELEASE.jar:4.1.9.RELEASE] > at > org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>( > ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18] > at > org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>( > ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18] > at > org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>( > XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1] > at > org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext( > XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1] > at > org.apache.activemq.xbean.XBeanBrokerFactory.createBroker( > XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1] > at > org.apache.activemq.broker.BrokerFactory.createBroker( > BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1] > at > org.apache.activemq.broker.BrokerFactory.createBroker( > BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.StartCommand.runTask( > StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.AbstractCommand. > execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.ShellCommand.runTask( > ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.AbstractCommand. > execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.ShellCommand.main( > ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method)[:1.8.0_102] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62)[:1.8.0_102] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] > at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] > at > org.apache.activemq.console.Main.runTaskClass(Main.java: > 262)[activemq.jar:5.14.1] > at > org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method)[:1.8.0_102] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62)[:1.8.0_102] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] > at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] > at > org.tanukisoftware.wrapper.WrapperSimpleApp.run( > WrapperSimpleApp.java:240)[wrapper.jar:3.2.3] > at java.lang.Thread.run(Thread.java:745)[:1.8.0_102] > 2017-02-03 13:22:58,211 | WARN | Exception encountered during context > initialization - cancelling refresh attempt: > org.springframework.beans.factory.BeanCreationException: Error creating > bean > with name 'invokeStart' defined in class path resource [jetty.xml]: > Invocation of init method failed; nested exception is > java.net.BindException: Address already in use | > org.apache.activemq.xbean.XBeanBrokerFactory$1 | WrapperSimpleAppMain > 2017-02-03 13:22:58,214 | DEBUG | JspServlet.destroy() | > org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain > 2017-02-03 13:22:58,217 | INFO | Destroying Spring FrameworkServlet > 'dispatcher' | /admin | WrapperSimpleAppMain > 2017-02-03 13:22:58,218 | DEBUG | JspServlet.destroy() | > org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain > 2017-02-03 13:22:58,222 | INFO | Apache ActiveMQ 5.14.1 (jdbcBroker2, > ID:mtl-nes-dev-web-48232-1486146033598-0:2) is shutting down | > org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain > 2017-02-03 13:22:58,223 | DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=jdbcBroker2, > connector=clientConnectors,connectorName=default > | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain > 2017-02-03 13:22:58,225 | TRACE | Execute[ActiveMQ > BrokerService[jdbcBroker2] Task] runnable: > org.apache.activemq.broker.TransportConnection$4@7af5520a | > org.apache.activemq.thread.TaskRunnerFactory | WrapperSimpleAppMain > 2017-02-03 13:22:58,225 | TRACE | Created and running thread[ActiveMQ > BrokerService[jdbcBroker2] Task-2]: Thread[ActiveMQ > BrokerService[jdbcBroker2] Task-2,5,main] | > org.apache.activemq.thread.TaskRunnerFactory | WrapperSimpleAppMain > 2017-02-03 13:22:58,225 | DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=jdbcBroker2, > connector=clientConnectors,connectorName=default, > connectionViewType=clientId,connectionName=mtl-nes-dev-cn3 > | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain > 2017-02-03 13:22:58,225 | DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=jdbcBroker2, > connector=clientConnectors,connectorName=default,connectionViewType= > remoteAddress,connectionName=tcp_//10.3.22.198_41192 > | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain > 2017-02-03 13:22:58,225 | DEBUG | Stopping connection: > tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection | > ActiveMQ BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,227 | DEBUG | Stopping transport > tcp:///10.3.22.198:41192@61616 | > org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,227 | DEBUG | Initialized TaskRunnerFactory[ActiveMQ > Task] using ExecutorService: null | > org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,227 | TRACE | Execute[ActiveMQ Task] runnable: > org.apache.activemq.transport.tcp.TcpTransport$1@7a152e97 | > org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,227 | TRACE | Created and running thread[ActiveMQ > Task-1]: Thread[ActiveMQ Task-1,5,main] | > org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,227 | TRACE | Closing socket > Socket[addr=/10.3.22.198,port=41192,localport=61616] | > org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1 > 2017-02-03 13:22:58,228 | DEBUG | Closed socket > Socket[addr=/10.3.22.198,port=41192,localport=61616] | > org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1 > 2017-02-03 13:22:58,228 | DEBUG | Stopped transport: tcp:// > 10.3.22.198:41192 > | org.apache.activemq.broker.TransportConnection | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,228 | TRACE | Shutdown timeout: 1 task: Transport > Connection to: tcp://10.3.22.198:41192 | > org.apache.activemq.thread.DedicatedTaskRunner | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,228 | TRACE | Run task done: Transport Connection to: > tcp://10.3.22.198:41192 | org.apache.activemq.thread.DedicatedTaskRunner | > ActiveMQ Connection Dispatcher: tcp://10.3.22.198:41192 > 2017-02-03 13:22:58,230 | DEBUG | Cleaning up connection resources: > tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection | > ActiveMQ BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,230 | DEBUG | remove connection id: > ID:mtl-nes-dev-cn3-38468-1486139024804-1:1 | > org.apache.activemq.broker.TransportConnection | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,230 | DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=jdbcBroker2, > destinationType=Queue,destinationName=ComputeNodeQueue-Alin, > endpoint=Consumer,clientId=mtl-nes-dev-cn3,consumerId=ID_ > mtl-nes-dev-cn3-38468-1486139024804-1_1_1_1 > | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,230 | DEBUG | jdbcBroker2 removing consumer: > ID:mtl-nes-dev-cn3-38468-1486139024804-1:1:1:1 for destination: > queue://ComputeNodeQueue-Alin | > org.apache.activemq.broker.region.AbstractRegion | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,234 | DEBUG | queue://ComputeNodeQueue-Alin remove sub: > QueueSubscription: consumer=ID:mtl-nes-dev-cn3- > 38468-1486139024804-1:1:1:1, > destinations=1, dispatched=0, delivered=0, pending=0, prefetch=1, > prefetchExtension=0, lastDeliveredSeqId: -2, dequeues: 0, dispatched: 0, > inflight: 0, groups: 0 | org.apache.activemq.broker.region.Queue | > ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,235 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616 > for broker transport URI: tcp://mtl-nes-dev-web:61616 | > org.apache.activemq.broker.TransportConnector | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,235 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616 > for broker transport URI: tcp://mtl-nes-dev-web:61616 | > org.apache.activemq.broker.TransportConnector | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,236 | DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=jdbcBroker2, > destinationType=Topic,destinationName=ActiveMQ. > Advisory.TempQueue_ActiveMQ.Advisory.TempTopic,endpoint= > Consumer,clientId=mtl-nes-dev-cn3,consumerId=ID_mtl-nes-dev- > cn3-38468-1486139024804-1_1_-1_1 > | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,236 | DEBUG | jdbcBroker2 removing consumer: > ID:mtl-nes-dev-cn3-38468-1486139024804-1:1:-1:1 for destination: > ActiveMQ.Advisory.TempQueue,ActiveMQ.Advisory.TempTopic | > org.apache.activemq.broker.region.AbstractRegion | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,237 | TRACE | Running task > queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0, > pending=0 > | org.apache.activemq.thread.DedicatedTaskRunner | > Queue:ComputeNodeQueue-Alin > 2017-02-03 13:22:58,237 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616 > for broker transport URI: tcp://mtl-nes-dev-web:61616 | > org.apache.activemq.broker.TransportConnector | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,237 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616 > for broker transport URI: tcp://mtl-nes-dev-web:61616 | > org.apache.activemq.broker.TransportConnector | ActiveMQ > BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,237 | DEBUG | queue://ComputeNodeQueue-Alin, > subscriptions=0, memory=0%, size=0, pending=0 toPageIn: 0, force:false, > Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0, > enqueueCount: 0, dequeueCount: 0, memUsage:0, maxPageSize:200 | > org.apache.activemq.broker.region.Queue | Queue:ComputeNodeQueue-Alin > 2017-02-03 13:22:58,238 | DEBUG | Connection Stopped: > tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection | > ActiveMQ BrokerService[jdbcBroker2] Task-2 > 2017-02-03 13:22:58,238 | INFO | Connector default stopped | > org.apache.activemq.broker.TransportConnector | WrapperSimpleAppMain > 2017-02-03 13:22:58,239 | TRACE | Shutdown timeout: 0 task: > queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0, > pending=0 > | org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain > 2017-02-03 13:22:58,240 | TRACE | Run task done: > queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0, > pending=0 > | org.apache.activemq.thread.DedicatedTaskRunner | > Queue:ComputeNodeQueue-Alin > 2017-02-03 13:22:58,241 | TRACE | ComputeNodeQueue-Alin,pendingSize:0 > resetBatching. last recovered: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1] | > org.apache.activemq.store.jdbc.JDBCMessageStore | WrapperSimpleAppMain > 2017-02-03 13:22:58,241 | TRACE | Shutdown timeout: 0 task: Topic: > destination=ActiveMQ.Advisory.MasterBroker, subscriptions=0 | > org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain > 2017-02-03 13:22:58,242 | TRACE | Shutdown timeout: 0 task: Topic: > destination=ActiveMQ.Advisory.Connection, subscriptions=0 | > org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain > 2017-02-03 13:22:58,242 | TRACE | Run task done: Topic: > destination=ActiveMQ.Advisory.MasterBroker, subscriptions=0 | > org.apache.activemq.thread.DedicatedTaskRunner | Topic > ActiveMQ.Advisory.MasterBroker > 2017-02-03 13:22:58,242 | TRACE | Run task done: Topic: > destination=ActiveMQ.Advisory.Connection, subscriptions=0 | > org.apache.activemq.thread.DedicatedTaskRunner | Topic > ActiveMQ.Advisory.Connection > 2017-02-03 13:22:58,242 | TRACE | Shutdown timeout: 0 task: Topic: > destination=ActiveMQ.Advisory.Queue, subscriptions=0 | > org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain > 2017-02-03 13:22:58,243 | TRACE | Run task done: Topic: > destination=ActiveMQ.Advisory.Queue, subscriptions=0 | > org.apache.activemq.thread.DedicatedTaskRunner | Topic > ActiveMQ.Advisory.Queue > 2017-02-03 13:22:58,243 | TRACE | Shutdown timeout: 0 task: Topic: > destination=ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin, > subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | > WrapperSimpleAppMain > 2017-02-03 13:22:58,243 | TRACE | Run task done: Topic: > destination=ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin, > subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | Topic > ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin > 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=jdbcBroker2, > destinationType=Topic,destinationName=ActiveMQ.Advisory.MasterBroker > | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain > 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=jdbcBroker2, > destinationType=Topic,destinationName=ActiveMQ.Advisory.Connection > | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain > 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=jdbcBroker2, > destinationType=Queue,destinationName=ComputeNodeQueue-Alin > | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain > 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=jdbcBroker2, > destinationType=Topic,destinationName=ActiveMQ.Advisory.Queue > | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain > 2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=jdbcBroker2, > destinationType=Topic,destinationName=ActiveMQ.Advisory.Consumer.Queue. > ComputeNodeQueue-Alin > | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain > 2017-02-03 13:22:58,243 | INFO | > PListStore:[/usr/local/Nuance/activemq/bin/linux-x86-64/ > activemq-data/jdbcBroker2/tmp_storage] > stopped | org.apache.activemq.store.kahadb.plist.PListStoreImpl | > WrapperSimpleAppMain > 2017-02-03 13:22:58,287 | INFO | jdbcBroker2, released lease | > org.apache.activemq.store.jdbc.LeaseDatabaseLocker | WrapperSimpleAppMain > 2017-02-03 13:22:58,287 | TRACE | Shutdown of ExecutorService: > java.util.concurrent.ScheduledThreadPoolExecutor@21e82532[Running, pool > size > = 4, active threads = 0, queued tasks = 2, completed tasks = 2] with await > termination: 0 millis | org.apache.activemq.util.ThreadPoolUtils | > WrapperSimpleAppMain > 2017-02-03 13:22:58,288 | DEBUG | Shutdown of ExecutorService: > java.util.concurrent.ScheduledThreadPoolExecutor@21e82532[Shutting down, > pool size = 4, active threads = 0, queued tasks = 0, completed tasks = 2] > is > shutdown: true and terminated: false took: 0.001 seconds. | > org.apache.activemq.util.ThreadPoolUtils | WrapperSimpleAppMain > 2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=jdbcBroker2,service=Health | > org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain > 2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=jdbcBroker2, > service=Log4JConfiguration > | org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain > 2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean > org.apache.activemq:type=Broker,brokerName=jdbcBroker2 | > org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain > 2017-02-03 13:22:58,288 | DEBUG | Stopping jmx connector | > org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain > 2017-02-03 13:22:58,288 | DEBUG | InitialContextFactory. > getInitialContext() > | jndi | WrapperSimpleAppMain > 2017-02-03 13:22:58,288 | DEBUG | Created initial context delegate for > local > namespace:org.eclipse.jetty.jndi.local.localContextRoot@574009c8 | jndi | > WrapperSimpleAppMain > 2017-02-03 13:22:58,292 | DEBUG | Unexported JMX RMI Registry | > org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain > 2017-02-03 13:22:58,292 | DEBUG | Forcing shutdown of ExecutorService: > java.util.concurrent.ThreadPoolExecutor@68f23545[Running, pool size = 0, > active threads = 0, queued tasks = 0, completed tasks = 0] | > org.apache.activemq.util.ThreadPoolUtils | WrapperSimpleAppMain > 2017-02-03 13:22:58,292 | TRACE | Shutdown of ExecutorService: > java.util.concurrent.ThreadPoolExecutor@68f23545[Terminated, pool size = > 0, > active threads = 0, queued tasks = 0, completed tasks = 0] is shutdown: > true > and terminated: true. | org.apache.activemq.util.ThreadPoolUtils | > WrapperSimpleAppMain > 2017-02-03 13:22:58,292 | INFO | Apache ActiveMQ 5.14.1 (jdbcBroker2, > ID:mtl-nes-dev-web-48232-1486146033598-0:2) uptime 1 minute | > org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain > 2017-02-03 13:22:58,292 | INFO | Apache ActiveMQ 5.14.1 (jdbcBroker2, > ID:mtl-nes-dev-web-48232-1486146033598-0:2) is shutdown | > org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain > 2017-02-03 13:22:58,295 | ERROR | Failed to load: class path resource > [activemq.xml], reason: Error creating bean with name 'invokeStart' defined > in class path resource [jetty.xml]: Invocation of init method failed; > nested > exception is java.net.BindException: Address already in use | > org.apache.activemq.xbean.XBeanBrokerFactory | WrapperSimpleAppMain > org.springframework.beans.factory.BeanCreationException: Error creating > bean > with name 'invokeStart' defined in class path resource [jetty.xml]: > Invocation of init method failed; nested exception is > java.net.BindException: Address already in use > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.initializeBean(AbstractAutowireCapableBeanFac > tory.java:1572)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.doCreateBean(AbstractAutowireCapableBeanFac > tory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac > tory.createBean(AbstractAutowireCapableBeanFac > tory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractBeanFactory$1. > getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9. > RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry. > getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9. > RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean( > AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE. > jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.AbstractBeanFactory.getBean( > AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE. > jar:4.1.9.RELEASE] > at > org.springframework.beans.factory.support.DefaultListableBeanFactory. > preInstantiateSingletons(DefaultListableBeanFactory. > java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.context.support.AbstractApplicationContext. > finishBeanFactoryInitialization(AbstractApplicationContext. > java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE] > at > org.springframework.context.support.AbstractApplicationContext.refresh( > AbstractApplicationContext.java:480)[spring-context-4.1. > 9.RELEASE.jar:4.1.9.RELEASE] > at > org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>( > ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18] > at > org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>( > ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18] > at > org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>( > XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1] > at > org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext( > XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1] > at > org.apache.activemq.xbean.XBeanBrokerFactory.createBroker( > XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1] > at > org.apache.activemq.broker.BrokerFactory.createBroker( > BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1] > at > org.apache.activemq.broker.BrokerFactory.createBroker( > BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.StartCommand.runTask( > StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.AbstractCommand. > execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.ShellCommand.runTask( > ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.AbstractCommand. > execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1] > at > org.apache.activemq.console.command.ShellCommand.main( > ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method)[:1.8.0_102] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62)[:1.8.0_102] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] > at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] > at > org.apache.activemq.console.Main.runTaskClass(Main.java: > 262)[activemq.jar:5.14.1] > at > org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method)[:1.8.0_102] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62)[:1.8.0_102] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43)[:1.8.0_102] > at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102] > at > org.tanukisoftware.wrapper.WrapperSimpleApp.run( > WrapperSimpleApp.java:240)[wrapper.jar:3.2.3] > at java.lang.Thread.run(Thread.java:745)[:1.8.0_102] > > > > > -- > View this message in context: http://activemq.2283324.n4. > nabble.com/Failover-problems-when-DB-replication-is-being- > used-tp4721586.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
