Shutdown fails if DB connection is lost
---------------------------------------
Key: AMQ-1352
URL: https://issues.apache.org/activemq/browse/AMQ-1352
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.0.0
Reporter: Mario Siegenthaler
The broker does not shutdown cleanly if the db connection for the JDBC
persistence gets disconnected.
I discovered this issue when implementing the fix for the AMQ-1350. When the
broker discovers it doesn't hold the db-lock anymore (because the connection is
lost, killed via the db-administration tool) it initiates its shutdown. This
shutdown fails most of the time, especially if there is a camel-rule or a
webconsole started within the broker.
The thing is that the broker actually has been shut down (as it's stated and
intended), but the other things (jetty, commandAgent, camel etc.) will not be
shut down and the java-process does not terminate.
IMO the broker shutdown didn't even fail, it's just issued an
InterruptedException somewhere in order to stop some connector thread. I think
this just needs a proper exception handling/ignoring.
The brokers console output:
INFO JDBCPersistenceAdapter - No longer able to keep the exclusive
lock so giving up being a master
INFO BrokerService - ActiveMQ Message Broker (marios-master,
ID:mario-siegenthalers-computer.local-64808-1185663859419-1:0) is shutting down
WARN ActiveMQConnection - Async exception with no exception
listener: java.io.EOFException
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:356)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:156)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:136)
at java.lang.Thread.run(Thread.java:637)
INFO AdvisoryConsumer - Failed to send remove command:
javax.jms.JMSException: The transport is not running.
javax.jms.JMSException: The transport is not running.
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:58)
at
org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1205)
at
org.apache.activemq.AdvisoryConsumer.dispose(AdvisoryConsumer.java:55)
at
org.apache.activemq.ActiveMQConnection.cleanup(ActiveMQConnection.java:1378)
at
org.apache.activemq.ActiveMQConnection.transportFailed(ActiveMQConnection.java:2074)
at
org.apache.activemq.ActiveMQConnection$4.run(ActiveMQConnection.java:1708)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:679)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:704)
at java.lang.Thread.run(Thread.java:637)
Caused by: java.io.IOException: The transport is not running.
at
org.apache.activemq.transport.TransportSupport.checkStarted(TransportSupport.java:104)
at
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:117)
at
org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:147)
at
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:80)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:93)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:44)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:56)
at
org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1203)
... 7 more
INFO TransportConnector - Connector openwire Stopped
ERROR efaultMessageListenerContainer - Setup of JMS message listener invoker
failed - trying to recover
javax.jms.IllegalStateException: The Consumer is closed
at
org.apache.activemq.ActiveMQMessageConsumer.checkClosed(ActiveMQMessageConsumer.java:650)
at
org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:484)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:375)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:300)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:254)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:871)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:818)
at java.lang.Thread.run(Thread.java:637)
ERROR ManagedQueueRegion - Could not stop service: QueueRegion:
destinations=1, subscriptions=0, memory=0%. Reason:
java.lang.InterruptedException
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at
org.apache.activemq.thread.DedicatedTaskRunner.shutdown(DedicatedTaskRunner.java:73)
at
org.apache.activemq.thread.DedicatedTaskRunner.shutdown(DedicatedTaskRunner.java:83)
at org.apache.activemq.broker.region.Queue.stop(Queue.java:565)
at
org.apache.activemq.broker.region.AbstractRegion.stop(AbstractRegion.java:94)
at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:42)
at
org.apache.activemq.broker.region.RegionBroker.doStop(RegionBroker.java:582)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.doStop(ManagedRegionBroker.java:112)
at
org.apache.activemq.broker.region.RegionBroker.stop(RegionBroker.java:196)
at
org.apache.activemq.broker.TransactionBroker.stop(TransactionBroker.java:115)
at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:158)
at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:158)
at
org.apache.activemq.broker.MutableBrokerFilter.stop(MutableBrokerFilter.java:168)
at
org.apache.activemq.broker.BrokerService$2.stop(BrokerService.java:1353)
at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:42)
at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:471)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.stopBroker(JDBCPersistenceAdapter.java:481)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:473)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$3.run(JDBCPersistenceAdapter.java:268)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:301)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:679)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:704)
at java.lang.Thread.run(Thread.java:637)
ERROR BrokerService$2 - Could not stop service: [EMAIL
PROTECTED] Reason: java.lang.InterruptedException
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at
org.apache.activemq.thread.DedicatedTaskRunner.shutdown(DedicatedTaskRunner.java:73)
at
org.apache.activemq.thread.DedicatedTaskRunner.shutdown(DedicatedTaskRunner.java:83)
at org.apache.activemq.broker.region.Queue.stop(Queue.java:565)
at
org.apache.activemq.broker.region.AbstractRegion.stop(AbstractRegion.java:94)
at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:42)
at
org.apache.activemq.broker.region.RegionBroker.doStop(RegionBroker.java:582)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.doStop(ManagedRegionBroker.java:112)
at
org.apache.activemq.broker.region.RegionBroker.stop(RegionBroker.java:196)
at
org.apache.activemq.broker.TransactionBroker.stop(TransactionBroker.java:115)
at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:158)
at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:158)
at
org.apache.activemq.broker.MutableBrokerFilter.stop(MutableBrokerFilter.java:168)
at
org.apache.activemq.broker.BrokerService$2.stop(BrokerService.java:1353)
at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:42)
at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:471)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.stopBroker(JDBCPersistenceAdapter.java:481)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:473)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$3.run(JDBCPersistenceAdapter.java:268)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:301)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:679)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:704)
at java.lang.Thread.run(Thread.java:637)
INFO BrokerService - ActiveMQ JMS Message Broker
(marios-master, ID:mario-siegenthalers-computer.local-64808-1185663859419-1:0)
stopped
WARN JDBCPersistenceAdapter - Failed to stop broker
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at
org.apache.activemq.thread.DedicatedTaskRunner.shutdown(DedicatedTaskRunner.java:73)
at
org.apache.activemq.thread.DedicatedTaskRunner.shutdown(DedicatedTaskRunner.java:83)
at org.apache.activemq.broker.region.Queue.stop(Queue.java:565)
at
org.apache.activemq.broker.region.AbstractRegion.stop(AbstractRegion.java:94)
at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:42)
at
org.apache.activemq.broker.region.RegionBroker.doStop(RegionBroker.java:582)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.doStop(ManagedRegionBroker.java:112)
at
org.apache.activemq.broker.region.RegionBroker.stop(RegionBroker.java:196)
at
org.apache.activemq.broker.TransactionBroker.stop(TransactionBroker.java:115)
at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:158)
at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:158)
at
org.apache.activemq.broker.MutableBrokerFilter.stop(MutableBrokerFilter.java:168)
at
org.apache.activemq.broker.BrokerService$2.stop(BrokerService.java:1353)
at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:42)
at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:471)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.stopBroker(JDBCPersistenceAdapter.java:481)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:473)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$3.run(JDBCPersistenceAdapter.java:268)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:301)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:679)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:704)
at java.lang.Thread.run(Thread.java:637)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
