[
https://issues.apache.org/jira/browse/ARTEMIS-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy A. Bish resolved ARTEMIS-5377.
--------------------------------------
Fix Version/s: 2.42.0
Resolution: Fixed
> Deadlock and Crash of Artemis Broker When Restarting a Bridged Peer
> -------------------------------------------------------------------
>
> Key: ARTEMIS-5377
> URL: https://issues.apache.org/jira/browse/ARTEMIS-5377
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker, Clustering
> Affects Versions: 2.31.2
> Reporter: Alexander
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.42.0
>
> Attachments: falled_broker.log, restarder_boker.log
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> I encountered a similar issue as described on ARTEMIS-4114.
> Configuration:
> {code:xml}
> <ha-policy>
> <live-only>
> <scale-down>
> <connectors>
> <connector-ref>ART.GW.CLS1-connector</connector-ref>
> <connector-ref>ART.GW.CLS2-connector</connector-ref>
> <connector-ref>ART.GW.CLS3-connector</connector-ref>
> </connectors>
> </scale-down>
> </live-only>
> </ha-policy>
> ...
> <bridges>
> <bridge name="ART.EL.CLS">
> <queue-name>ART.EL.CLS</queue-name>
> <forwarding-address>ART.EL.CLS</forwarding-address>
> <reconnect-attempts>-1</reconnect-attempts>
> <static-connectors>
> <connector-ref>ART.EL.CLS1-connector</connector-ref>
> <connector-ref>ART.EL.CLS2-connector</connector-ref>
> <connector-ref>ART.EL.CLS3-connector</connector-ref>
> <connector-ref>ART.EL.CLS4-connector</connector-ref>
> <connector-ref>ART.EL.CLS5connector</connector-ref>
> <connector-ref>ART.EL.CLS6-connector</connector-ref>
> </static-connectors>
> </bridge>
> ... {code}
>
> When restarting one broker, another broker connected to it via a bridge
> crashes with a deadlock error. Before the crash, the following warnings
> appeared on the connected broker:
> {noformat}
> 2025-03-08 02:39:09,089 WARN [org.apache.activemq.artemis.core.server]
> AMQ222094: Bridge unable to send message
> Reference[314223917984]:NON-RELIABLE:CoreMessage[...]@2117085983, will try
> again once bridge reconnects
> ActiveMQObjectClosedException[errorType=OBJECT_CLOSED message=AMQ219018:
> Producer is closed]
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.checkClosed(ClientProducerImpl.java:310)
> at
> org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:127)
> at
> org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.deliverStandardMessage(BridgeImpl.java:767)
> at
> org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:628)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.handle(QueueImpl.java:4055)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:3191)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4380)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
> 2025-03-08 02:39:09,089 WARN [org.apache.activemq.artemis.core.server]
> AMQ222095: Connection failed with failedOver=false
> 2025-03-08 02:39:09,089 WARN
> [org.apache.activemq.artemis.core.server.impl.QueueImpl] null
> java.util.NoSuchElementException
> at
> org.apache.activemq.artemis.utils.collections.PriorityLinkedListImpl$PriorityLinkedListIterator.repeat(PriorityLinkedListImpl.java:225)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:3214)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4380)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118){noformat}
>
> I have attached the full log. From it, you can see that at 02:41:21 Broker 1
> completely shut down. Then, at 02:41:38 Broker 2 also stopped with the
> following error:
> {noformat}
> 2025-03-08 02:41:38,069 INFO [org.apache.activemq.artemis.core.server]
> AMQ224107: The Critical Analyzer detected slow paths on the broker. It is
> recommended that you enable trace logs on
> org.apache.activemq.artemis.utils.critical while you troubleshoot this issue.
> You should disable the trace logs when you have finished troubleshooting.
> 2025-03-08 02:41:38,069 ERROR [org.apache.activemq.artemis.core.server]
> AMQ224079: The process for the virtual machine will be killed, as component
> QueueImpl[name=ART.EL.CLS, postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=ART.GW.CLS7], temp=false]@3789b8fe is not
> responsive
> 2025-03-08 02:41:40,152 WARN [org.apache.activemq.artemis.core.server]
> AMQ222199: Thread dump:
> *******************************************************************************
> Complete Thread dump
> "Reference Handler" Id=2 RUNNABLE
> at
> [email protected]/java.lang.ref.Reference.waitForReferencePendingList(Native
> Method)
> at
> [email protected]/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
> at
> [email protected]/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
>
>
> "Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@279208e0
> at [email protected]/java.lang.Object.wait(Native Method)
> - waiting on java.lang.ref.ReferenceQueue$Lock@279208e0
> at
> [email protected]/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
> at
> [email protected]/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
> at
> [email protected]/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
>
>
> "Signal Dispatcher" Id=4 RUNNABLE
> ...{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information, visit: https://activemq.apache.org/contact
