Hi,
In a activemq clustered environment (Mysql DB persisted) whenever there is
DB connection glitch (DB is not available for a couple of seconds) at least
1 activemq cluster node gets into a weird state, which eventually will
cause the termination of the activemq process. This his how i reproduce this
behavior:
1. I have an activemq cluster composed of 2 nodes: Node 1 and Node 2. The
messages persistence is done using a MYSQL instance all set and good here.
If there are no issue on the network the nodes are properly being activated
one at a time with the second node taking over when the first one fails.
<broker brokerName="node1"
xmlns="http://activemq.apache.org/schema/core";>
<persistenceAdapter>
<jdbcPersistenceAdapter lockKeepAlivePeriod="1000"
lockAcquireSleepInterval="2000" dataSource="#mysql-ds">
<locker>
<lease-database-locker lockAcquireSleepInterval="2000"/>
</locker>
<statements>
<statements binaryDataType ="LONGBLOB"/>
</statements>
</jdbcPersistenceAdapter>
</persistenceAdapter>
<transportConnectors>
<transportConnector name="default" uri="tcp://0.0.0.0:61616"/>
</transportConnectors>
</broker>
<bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
2. Let's say that currently Node1 is the owner of the lock. At this moment
if i try to access the Node2 admin console i will not be able to get
anything in the browser window. But i'm able to get the console for Node1
3. If i mess with the DB network (to simulate a network glitch) #> service
network restart. If i'm lucky enough Node1 will loose the key and Node2 will
become master. At this moment if i check again the admin consoles for the 2
nodes I can see that Node2 admin console is available, but Node1 console
will be available too. Is jsut that it will present an error on the page
(this behavior is different than the usual slave behavior.)
4. If at this point I stop the Node2 activemq service, I would have expected
Node1 to take over. unfortunately Node1 will crash at this point with some
java.net.BindException: Address already in use errors. You can find the logs
for the failing Node1 below.
Any idea what is going on here?
Thanks
2017-02-03 13:22:58,193 | DEBUG | Scratch dir for the JSP engine is:
/tmp/jetty-0.0.0.0-8161-api-_api-any-3039768726853340463.dir/jsp |
org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain
2017-02-03 13:22:58,193 | DEBUG | IMPORTANT: Do not modify the generated
servlets | org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain
2017-02-03 13:22:58,194 | DEBUG | Defaulting to use topics: true |
org.apache.activemq.web.MessageServletSupport | WrapperSimpleAppMain
2017-02-03 13:22:58,197 | INFO | jolokia-agent: Using policy access
restrictor classpath:/jolokia-access.xml | /api | WrapperSimpleAppMain
2017-02-03 13:22:58,203 | DEBUG | InitialContextFactory.getInitialContext()
| jndi | WrapperSimpleAppMain
2017-02-03 13:22:58,203 | DEBUG | Created initial context delegate for local
namespace:org.eclipse.jetty.jndi.local.localContextRoot@d0793bc | jndi |
WrapperSimpleAppMain
2017-02-03 13:22:58,203 | DEBUG | Looking up name="comp/env/jmx/runtime" |
jndi | WrapperSimpleAppMain
2017-02-03 13:22:58,203 | DEBUG | Trying thread context classloader | jndi |
WrapperSimpleAppMain
2017-02-03 13:22:58,203 | DEBUG | supportDeepBinding=false | jndi |
WrapperSimpleAppMain
2017-02-03 13:22:58,203 | DEBUG | Made context comp for classloader:
WebAppClassLoader=Apache ActiveMQ REST API@2c29a88d | jndi |
WrapperSimpleAppMain
2017-02-03 13:22:58,203 | DEBUG | Looking up name="env/jmx/runtime" | jndi |
WrapperSimpleAppMain
2017-02-03 13:22:58,207 | INFO | jolokia-agent: jolokia:type=Config is
already registered. Adding it with
jolokia:type=Config,uuid=a5975fe0-a299-4be1-85c0-7431ef20c86c, but you
should revise your setup in order to either use a qualifier or ensure, that
only a single agent gets registered (otherwise history functionality might
not work) | /api | WrapperSimpleAppMain
2017-02-03 13:22:58,207 | INFO | jolokia-agent: Cannot register (legacy)
MBean handler for config store with name jmx4perl:type=Config since it
already exists. This is the case if another agent has been already started
within the same JVM. The registration is skipped. | /api |
WrapperSimpleAppMain
2017-02-03 13:22:58,207 | INFO | jolokia-agent: Jolokia Discovery MBean
registration is skipped because there is already one registered. | /api |
WrapperSimpleAppMain
2017-02-03 13:22:58,208 | WARN | FAILED
ServerConnector@1175f42f{HTTP/1.1}{0.0.0.0:8161}: java.net.BindException:
Address already in use | org.eclipse.jetty.util.component.AbstractLifeCycle
| WrapperSimpleAppMain
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)[:1.8.0_102]
at sun.nio.ch.Net.bind(Net.java:433)[:1.8.0_102]
at sun.nio.ch.Net.bind(Net.java:425)[:1.8.0_102]
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)[:1.8.0_102]
at
sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)[:1.8.0_102]
at
org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.Server.doStart(Server.java:366)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)[:1.8.0_102]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
at
org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)[spring-core-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.config.MethodInvokingBean.invokeWithTargetException(MethodInvokingBean.java:119)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.config.MethodInvokingFactoryBean.afterPropertiesSet(MethodInvokingFactoryBean.java:106)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1631)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1568)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18]
at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18]
at
org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
at
org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
at
org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1]
at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1]
at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)[:1.8.0_102]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
at
org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.14.1]
at
org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)[:1.8.0_102]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
at
org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)[wrapper.jar:3.2.3]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_102]
2017-02-03 13:22:58,210 | WARN | FAILED
org.eclipse.jetty.server.Server@5f70f898: java.net.BindException: Address
already in use | org.eclipse.jetty.util.component.AbstractLifeCycle |
WrapperSimpleAppMain
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)[:1.8.0_102]
at sun.nio.ch.Net.bind(Net.java:433)[:1.8.0_102]
at sun.nio.ch.Net.bind(Net.java:425)[:1.8.0_102]
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)[:1.8.0_102]
at
sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)[:1.8.0_102]
at
org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.server.Server.doStart(Server.java:366)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)[jetty-all-9.2.13.v20150730.jar:9.2.13.v20150730]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)[:1.8.0_102]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
at
org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)[spring-core-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.config.MethodInvokingBean.invokeWithTargetException(MethodInvokingBean.java:119)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.config.MethodInvokingFactoryBean.afterPropertiesSet(MethodInvokingFactoryBean.java:106)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1631)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1568)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18]
at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18]
at
org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
at
org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
at
org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1]
at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1]
at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)[:1.8.0_102]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
at
org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.14.1]
at
org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)[:1.8.0_102]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
at
org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)[wrapper.jar:3.2.3]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_102]
2017-02-03 13:22:58,211 | WARN | Exception encountered during context
initialization - cancelling refresh attempt:
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'invokeStart' defined in class path resource [jetty.xml]:
Invocation of init method failed; nested exception is
java.net.BindException: Address already in use |
org.apache.activemq.xbean.XBeanBrokerFactory$1 | WrapperSimpleAppMain
2017-02-03 13:22:58,214 | DEBUG | JspServlet.destroy() |
org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain
2017-02-03 13:22:58,217 | INFO | Destroying Spring FrameworkServlet
'dispatcher' | /admin | WrapperSimpleAppMain
2017-02-03 13:22:58,218 | DEBUG | JspServlet.destroy() |
org.apache.jasper.servlet.JspServlet | WrapperSimpleAppMain
2017-02-03 13:22:58,222 | INFO | Apache ActiveMQ 5.14.1 (jdbcBroker2,
ID:mtl-nes-dev-web-48232-1486146033598-0:2) is shutting down |
org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
2017-02-03 13:22:58,223 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=jdbcBroker2,connector=clientConnectors,connectorName=default
| org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,225 | TRACE | Execute[ActiveMQ
BrokerService[jdbcBroker2] Task] runnable:
org.apache.activemq.broker.TransportConnection$4@7af5520a |
org.apache.activemq.thread.TaskRunnerFactory | WrapperSimpleAppMain
2017-02-03 13:22:58,225 | TRACE | Created and running thread[ActiveMQ
BrokerService[jdbcBroker2] Task-2]: Thread[ActiveMQ
BrokerService[jdbcBroker2] Task-2,5,main] |
org.apache.activemq.thread.TaskRunnerFactory | WrapperSimpleAppMain
2017-02-03 13:22:58,225 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=jdbcBroker2,connector=clientConnectors,connectorName=default,connectionViewType=clientId,connectionName=mtl-nes-dev-cn3
| org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,225 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=jdbcBroker2,connector=clientConnectors,connectorName=default,connectionViewType=remoteAddress,connectionName=tcp_//10.3.22.198_41192
| org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,225 | DEBUG | Stopping connection:
tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection |
ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,227 | DEBUG | Stopping transport
tcp:///10.3.22.198:41192@61616 |
org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,227 | DEBUG | Initialized TaskRunnerFactory[ActiveMQ
Task] using ExecutorService: null |
org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,227 | TRACE | Execute[ActiveMQ Task] runnable:
org.apache.activemq.transport.tcp.TcpTransport$1@7a152e97 |
org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,227 | TRACE | Created and running thread[ActiveMQ
Task-1]: Thread[ActiveMQ Task-1,5,main] |
org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,227 | TRACE | Closing socket
Socket[addr=/10.3.22.198,port=41192,localport=61616] |
org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1
2017-02-03 13:22:58,228 | DEBUG | Closed socket
Socket[addr=/10.3.22.198,port=41192,localport=61616] |
org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1
2017-02-03 13:22:58,228 | DEBUG | Stopped transport: tcp://10.3.22.198:41192
| org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,228 | TRACE | Shutdown timeout: 1 task: Transport
Connection to: tcp://10.3.22.198:41192 |
org.apache.activemq.thread.DedicatedTaskRunner | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,228 | TRACE | Run task done: Transport Connection to:
tcp://10.3.22.198:41192 | org.apache.activemq.thread.DedicatedTaskRunner |
ActiveMQ Connection Dispatcher: tcp://10.3.22.198:41192
2017-02-03 13:22:58,230 | DEBUG | Cleaning up connection resources:
tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection |
ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,230 | DEBUG | remove connection id:
ID:mtl-nes-dev-cn3-38468-1486139024804-1:1 |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,230 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Queue,destinationName=ComputeNodeQueue-Alin,endpoint=Consumer,clientId=mtl-nes-dev-cn3,consumerId=ID_mtl-nes-dev-cn3-38468-1486139024804-1_1_1_1
| org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,230 | DEBUG | jdbcBroker2 removing consumer:
ID:mtl-nes-dev-cn3-38468-1486139024804-1:1:1:1 for destination:
queue://ComputeNodeQueue-Alin |
org.apache.activemq.broker.region.AbstractRegion | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,234 | DEBUG | queue://ComputeNodeQueue-Alin remove sub:
QueueSubscription: consumer=ID:mtl-nes-dev-cn3-38468-1486139024804-1:1:1:1,
destinations=1, dispatched=0, delivered=0, pending=0, prefetch=1,
prefetchExtension=0, lastDeliveredSeqId: -2, dequeues: 0, dispatched: 0,
inflight: 0, groups: 0 | org.apache.activemq.broker.region.Queue | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,235 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616
for broker transport URI: tcp://mtl-nes-dev-web:61616 |
org.apache.activemq.broker.TransportConnector | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,235 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616
for broker transport URI: tcp://mtl-nes-dev-web:61616 |
org.apache.activemq.broker.TransportConnector | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,236 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.TempQueue_ActiveMQ.Advisory.TempTopic,endpoint=Consumer,clientId=mtl-nes-dev-cn3,consumerId=ID_mtl-nes-dev-cn3-38468-1486139024804-1_1_-1_1
| org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,236 | DEBUG | jdbcBroker2 removing consumer:
ID:mtl-nes-dev-cn3-38468-1486139024804-1:1:-1:1 for destination:
ActiveMQ.Advisory.TempQueue,ActiveMQ.Advisory.TempTopic |
org.apache.activemq.broker.region.AbstractRegion | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,237 | TRACE | Running task
queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0, pending=0
| org.apache.activemq.thread.DedicatedTaskRunner |
Queue:ComputeNodeQueue-Alin
2017-02-03 13:22:58,237 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616
for broker transport URI: tcp://mtl-nes-dev-web:61616 |
org.apache.activemq.broker.TransportConnector | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,237 | DEBUG | Publishing: tcp://mtl-nes-dev-web:61616
for broker transport URI: tcp://mtl-nes-dev-web:61616 |
org.apache.activemq.broker.TransportConnector | ActiveMQ
BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,237 | DEBUG | queue://ComputeNodeQueue-Alin,
subscriptions=0, memory=0%, size=0, pending=0 toPageIn: 0, force:false,
Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0,
enqueueCount: 0, dequeueCount: 0, memUsage:0, maxPageSize:200 |
org.apache.activemq.broker.region.Queue | Queue:ComputeNodeQueue-Alin
2017-02-03 13:22:58,238 | DEBUG | Connection Stopped:
tcp://10.3.22.198:41192 | org.apache.activemq.broker.TransportConnection |
ActiveMQ BrokerService[jdbcBroker2] Task-2
2017-02-03 13:22:58,238 | INFO | Connector default stopped |
org.apache.activemq.broker.TransportConnector | WrapperSimpleAppMain
2017-02-03 13:22:58,239 | TRACE | Shutdown timeout: 0 task:
queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0, pending=0
| org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain
2017-02-03 13:22:58,240 | TRACE | Run task done:
queue://ComputeNodeQueue-Alin, subscriptions=0, memory=0%, size=0, pending=0
| org.apache.activemq.thread.DedicatedTaskRunner |
Queue:ComputeNodeQueue-Alin
2017-02-03 13:22:58,241 | TRACE | ComputeNodeQueue-Alin,pendingSize:0
resetBatching. last recovered: [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1] |
org.apache.activemq.store.jdbc.JDBCMessageStore | WrapperSimpleAppMain
2017-02-03 13:22:58,241 | TRACE | Shutdown timeout: 0 task: Topic:
destination=ActiveMQ.Advisory.MasterBroker, subscriptions=0 |
org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain
2017-02-03 13:22:58,242 | TRACE | Shutdown timeout: 0 task: Topic:
destination=ActiveMQ.Advisory.Connection, subscriptions=0 |
org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain
2017-02-03 13:22:58,242 | TRACE | Run task done: Topic:
destination=ActiveMQ.Advisory.MasterBroker, subscriptions=0 |
org.apache.activemq.thread.DedicatedTaskRunner | Topic
ActiveMQ.Advisory.MasterBroker
2017-02-03 13:22:58,242 | TRACE | Run task done: Topic:
destination=ActiveMQ.Advisory.Connection, subscriptions=0 |
org.apache.activemq.thread.DedicatedTaskRunner | Topic
ActiveMQ.Advisory.Connection
2017-02-03 13:22:58,242 | TRACE | Shutdown timeout: 0 task: Topic:
destination=ActiveMQ.Advisory.Queue, subscriptions=0 |
org.apache.activemq.thread.DedicatedTaskRunner | WrapperSimpleAppMain
2017-02-03 13:22:58,243 | TRACE | Run task done: Topic:
destination=ActiveMQ.Advisory.Queue, subscriptions=0 |
org.apache.activemq.thread.DedicatedTaskRunner | Topic
ActiveMQ.Advisory.Queue
2017-02-03 13:22:58,243 | TRACE | Shutdown timeout: 0 task: Topic:
destination=ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin,
subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner |
WrapperSimpleAppMain
2017-02-03 13:22:58,243 | TRACE | Run task done: Topic:
destination=ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin,
subscriptions=0 | org.apache.activemq.thread.DedicatedTaskRunner | Topic
ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin
2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.MasterBroker
| org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.Connection
| org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Queue,destinationName=ComputeNodeQueue-Alin
| org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.Queue
| org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,243 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=jdbcBroker2,destinationType=Topic,destinationName=ActiveMQ.Advisory.Consumer.Queue.ComputeNodeQueue-Alin
| org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,243 | INFO |
PListStore:[/usr/local/Nuance/activemq/bin/linux-x86-64/activemq-data/jdbcBroker2/tmp_storage]
stopped | org.apache.activemq.store.kahadb.plist.PListStoreImpl |
WrapperSimpleAppMain
2017-02-03 13:22:58,287 | INFO | jdbcBroker2, released lease |
org.apache.activemq.store.jdbc.LeaseDatabaseLocker | WrapperSimpleAppMain
2017-02-03 13:22:58,287 | TRACE | Shutdown of ExecutorService:
java.util.concurrent.ScheduledThreadPoolExecutor@21e82532[Running, pool size
= 4, active threads = 0, queued tasks = 2, completed tasks = 2] with await
termination: 0 millis | org.apache.activemq.util.ThreadPoolUtils |
WrapperSimpleAppMain
2017-02-03 13:22:58,288 | DEBUG | Shutdown of ExecutorService:
java.util.concurrent.ScheduledThreadPoolExecutor@21e82532[Shutting down,
pool size = 4, active threads = 0, queued tasks = 0, completed tasks = 2] is
shutdown: true and terminated: false took: 0.001 seconds. |
org.apache.activemq.util.ThreadPoolUtils | WrapperSimpleAppMain
2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=jdbcBroker2,service=Health |
org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=jdbcBroker2,service=Log4JConfiguration
| org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,288 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=jdbcBroker2 |
org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,288 | DEBUG | Stopping jmx connector |
org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,288 | DEBUG | InitialContextFactory.getInitialContext()
| jndi | WrapperSimpleAppMain
2017-02-03 13:22:58,288 | DEBUG | Created initial context delegate for local
namespace:org.eclipse.jetty.jndi.local.localContextRoot@574009c8 | jndi |
WrapperSimpleAppMain
2017-02-03 13:22:58,292 | DEBUG | Unexported JMX RMI Registry |
org.apache.activemq.broker.jmx.ManagementContext | WrapperSimpleAppMain
2017-02-03 13:22:58,292 | DEBUG | Forcing shutdown of ExecutorService:
java.util.concurrent.ThreadPoolExecutor@68f23545[Running, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 0] |
org.apache.activemq.util.ThreadPoolUtils | WrapperSimpleAppMain
2017-02-03 13:22:58,292 | TRACE | Shutdown of ExecutorService:
java.util.concurrent.ThreadPoolExecutor@68f23545[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 0] is shutdown: true
and terminated: true. | org.apache.activemq.util.ThreadPoolUtils |
WrapperSimpleAppMain
2017-02-03 13:22:58,292 | INFO | Apache ActiveMQ 5.14.1 (jdbcBroker2,
ID:mtl-nes-dev-web-48232-1486146033598-0:2) uptime 1 minute |
org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
2017-02-03 13:22:58,292 | INFO | Apache ActiveMQ 5.14.1 (jdbcBroker2,
ID:mtl-nes-dev-web-48232-1486146033598-0:2) is shutdown |
org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
2017-02-03 13:22:58,295 | ERROR | Failed to load: class path resource
[activemq.xml], reason: Error creating bean with name 'invokeStart' defined
in class path resource [jetty.xml]: Invocation of init method failed; nested
exception is java.net.BindException: Address already in use |
org.apache.activemq.xbean.XBeanBrokerFactory | WrapperSimpleAppMain
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'invokeStart' defined in class path resource [jetty.xml]:
Invocation of init method failed; nested exception is
java.net.BindException: Address already in use
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1572)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:736)[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)[spring-context-4.1.9.RELEASE.jar:4.1.9.RELEASE]
at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-3.18.jar:3.18]
at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-3.18.jar:3.18]
at
org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
at
org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.14.1.jar:5.14.1]
at
org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.14.1.jar:5.14.1]
at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.14.1.jar:5.14.1]
at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.14.1.jar:5.14.1]
at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.14.1.jar:5.14.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)[:1.8.0_102]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
at
org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.14.1]
at
org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.14.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)[:1.8.0_102]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_102]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_102]
at
org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)[wrapper.jar:3.2.3]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_102]
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Failover-problems-when-DB-replication-is-being-used-tp4721586.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
