Hi,
I am having an issue with failback to the master broker. I start the master
broker then the slave broker. I see the replication from the master to the
slave happen. I stop the master/active broker and a failover happens to the
slave as expected. When the master is started a failback to it is expected but
does not happen. The following is my ha-policy config. Did I miss something
in the ha-policy configuration? We are using version 2.28.0 and java17.
Thanks,
Paul
Master broker.xml config
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
Slave broker.xml config
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
The following are the logs from the master and slave brokers
HA “master” broker log after restart
2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
AMQ221020: Started EPOLL Acceptor at
cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
protocols [MQTT]
2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
AMQ221020: Started EPOLL Acceptor at
cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
protocols [CORE]
2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
AMQ221007: Server is now live
2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0 [amq-broker,
nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003: Starting
embedded web server
2023-12-08 20:42:57,625 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message
with ID: 15, result: NO_BINDINGS
2023-12-08 20:42:57,627 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created queue:
QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
temp=false]@15525ed3
2023-12-08 20:42:57,657 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2[ID=47edf16f<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2[ID=47edf16f>,
local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
2023-12-08 20:42:57,720 INFO
[org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
activemq-branding plugin
2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
temp=false]@15525ed3 targetConnector=ServerLocatorImpl
(identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
temp=false]@15525ed3 targetConnector=ServerLocatorImpl
[initialConnectors=[TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883[nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
connector=TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
address=, server=ActiveMQServerImpl::name=amq-broker]))
[initialConnectors=[TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
discoveryGroupConfiguration=null]] is connected
2023-12-08 20:42:57,755 INFO
[org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
artemis-plugin plugin
2023-12-08 20:42:57,766 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message
with ID: 18, result: NO_BINDINGS
2023-12-08 20:42:57,767 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message
with ID: 20, result: NO_BINDINGS
2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener] Initialising
hawtio services
2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager] Configuration will
be discovered via system properties
2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to Hawtio
2.15.0
2023-12-08 20:42:57,975 INFO [io.hawt.web.auth.AuthenticationConfiguration]
Starting hawtio authentication filter, JAAS realm: "activemq" authorized
role(s): "admin" role principal classes:
"org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter] Hawtio
loginRedirectFilter is using 1800 sec. HttpSession timeout
2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy servlet is
disabled
2023-12-08 20:42:57,989 INFO
[io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
property: [key=policyLocation,
value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001: HTTP
Server started at
http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002: Artemis
Jolokia REST API available at
http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
Console available at
http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
2023-12-08 20:42:58,455 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
connection: RemotingConnectionImpl [ID=f85ec51b, clientID=null,
nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d[ID=f85ec51b<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d[ID=f85ec51b>,
local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
2023-12-08 20:42:59,524 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858[ID=7cef23a2<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858[ID=7cef23a2>,
local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
2023-12-08 20:42:59,535 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message
with ID: 22, result: NO_BINDINGS
2023-12-08 20:42:59,536 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session connectionID:
7cef23a2
2023-12-08 20:42:59,549 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message
with ID: 24, result: NO_BINDINGS
2023-12-08 20:42:59,550 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session connectionID:
7cef23a2
2023-12-08 20:42:59,583 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message
with ID: 26, result: OK
2023-12-08 20:42:59,584 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created queue:
QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
temp=true]@5f46053b
2023-12-08 20:42:59,593 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
consumer with ID: 0, with session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7
2023-12-08 20:42:59,594 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message
with ID: 28, result: OK
2023-12-08 20:42:59,616 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent message
with ID: 29, result: OK, transaction: UNAVAILABLE
2023-12-08 20:42:59,615 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged
message:
Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
timestamp=0,expiration=0, durable=false,
address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@967407085,
with transaction: null
2023-12-08 20:42:59,621 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
message with message ID: 31, to consumer on address: activemq.notifications,
queue:
notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
2023-12-08 20:42:59,621 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged
message:
Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179613,_AMQ_ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
with transaction: null
2023-12-08 20:42:59,622 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
message with message ID: 32, to consumer on address: activemq.notifications,
queue:
notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
2023-12-08 20:42:59,622 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged
message:
Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179614,_AMQ_ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
with transaction: null
2023-12-08 20:42:59,622 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
message with message ID: 35, to consumer on address: activemq.notifications,
queue:
notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
2023-12-08 20:42:59,622 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged
message:
Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
timestamp=0,expiration=0, durable=false,
address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@333704293,
with transaction: null
2023-12-08 20:42:59,622 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
message with message ID: 39, to consumer on address: activemq.notifications,
queue:
notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
2023-12-08 20:43:08,447 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256[ID=1d1d79ce<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256[ID=1d1d79ce>,
local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
2023-12-08 20:43:08,464 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7[ID=8f88c6d5<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7[ID=8f88c6d5>,
local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
HA “slave” broker log.
2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null] started,
waiting live to fail before it gets active
2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is synchronized
with live server, nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
AMQ221031: backup announced
2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
AMQ221084: Requested 0 quorum votes
2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
AMQ221083: ignoring quorum vote as max cluster size is 1.
2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
AMQ221071: Failing over based on quorum vote results.
2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
AMQ212037: Connection failure to
cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/10.44.0.20:61616
has been detected: AMQ219015: The connection was disconnected because of
server shutdown [code=DISCONNECTED]
2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
AMQ212037: Connection failure to
cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/10.44.0.20:61616
has been detected: AMQ219015: The connection was disconnected because of
server shutdown [code=DISCONNECTED]
2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
AMQ221080: Deploying address DLQ supporting [ANYCAST]
2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
AMQ221007: Server is now live
2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
AMQ221020: Started EPOLL Acceptor at
cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
protocols [MQTT]
2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
AMQ221020: Started EPOLL Acceptor at
cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
protocols [CORE]
2023-12-08 20:42:51,946 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e>,
local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
2023-12-08 20:42:56,977 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001: destroyed
connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e>,
local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
AMQ212004: Failed to connect to server.
2023-12-08 20:42:57,445 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message
with ID: 26, result: NO_BINDINGS
2023-12-08 20:42:57,447 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created queue:
QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
temp=false]@737e64a6
2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
AMQ222186: unable to authorise cluster control: AMQ219016: Connection failure
detected. Unblocking a blocking call that will never get a response
2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
AMQ224091: Bridge ClusterConnectionBridge@2a063f52
[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
temp=false]@737e64a6 targetConnector=ServerLocatorImpl
(identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
temp=false]@737e64a6 targetConnector=ServerLocatorImpl
[initialConnectors=[TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008[nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
connector=TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
address=, server=ActiveMQServerImpl::name=amq-broker]))
[initialConnectors=[TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
discoveryGroupConfiguration=null]] is unable to connect to destination.
Retrying
2023-12-08 20:42:57,629 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251[ID=483170f6<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251[ID=483170f6>,
local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
2023-12-08 20:42:57,630 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b[ID=4514a2b3<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b[ID=4514a2b3>,
local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
2023-12-08 20:42:57,648 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8[ID=fcde4d78<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8[ID=fcde4d78>,
local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
2023-12-08 20:42:57,663 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message
with ID: 29, result: NO_BINDINGS
2023-12-08 20:42:57,663 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session connectionID:
fcde4d78
2023-12-08 20:42:57,679 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message
with ID: 31, result: NO_BINDINGS
2023-12-08 20:42:57,679 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session connectionID:
fcde4d78
2023-12-08 20:42:57,708 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message
with ID: 33, result: OK
2023-12-08 20:42:57,709 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created queue:
QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
temp=true]@3645037f
2023-12-08 20:42:57,719 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
consumer with ID: 0, with session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
2023-12-08 20:42:57,720 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message
with ID: 35, result: OK
2023-12-08 20:42:57,755 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent message
with ID: 36, result: OK, transaction: UNAVAILABLE
2023-12-08 20:42:57,753 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged
message:
Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
timestamp=0,expiration=0, durable=false,
address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@2125985563,
with transaction: null
2023-12-08 20:42:57,760 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
message with message ID: 38, to consumer on address: activemq.notifications,
queue:
notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
2023-12-08 20:42:57,761 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged
message:
Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
with transaction: null
2023-12-08 20:42:57,761 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
message with message ID: 41, to consumer on address: activemq.notifications,
queue:
notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
2023-12-08 20:42:57,762 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged
message:
Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
with transaction: null
2023-12-08 20:42:57,762 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
message with message ID: 43, to consumer on address: activemq.notifications,
queue:
notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
2023-12-08 20:42:57,762 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged
message:
Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
timestamp=0,expiration=0, durable=false,
address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@1828250555,
with transaction: null
2023-12-08 20:42:57,762 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
message with message ID: 44, to consumer on address: activemq.notifications,
queue:
notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
AMQ221027: Bridge ClusterConnectionBridge@2a063f52
[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
temp=false]@737e64a6 targetConnector=ServerLocatorImpl
(identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
temp=false]@737e64a6 targetConnector=ServerLocatorImpl
[initialConnectors=[TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008[nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
connector=TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
address=, server=ActiveMQServerImpl::name=amq-broker]))
[initialConnectors=[TransportConfiguration(name=artemis,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
discoveryGroupConfiguration=null]] is connected
2023-12-08 20:42:59,625 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message
with ID: 46, result: OK
2023-12-08 20:42:59,625 INFO
[org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message
with ID: 48, result: OK
2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
AMQ222186: unable to authorise cluster control: AMQ219016: Connection failure
detected. Unblocking a blocking call that will never get a response
