We are facing a strange issue while syncing messages from source to target broker cluster using Core bridge. Our source and target brokers are symmetric cluster and each has 2 nodes.
Since both source and target cluster has 2 nodes each, we have defined bridges from source cluster to target cluster as following. Idea is any source node can send messages to any available target node independently. This configuration is working fine most of the time but all of sudden we see some messages in delivering count on any of the source nodes. Bridge stat on that source node also shows a gap b/w 'Message Acknowledged' vs 'Message Pending Acknowledgment'. This gap is same as no. of messages showing in delivering count. Address stat on this queue also shows gap b/w 'Message Added' and 'Message Acknowledged'. This gap is same as no. of messages showing in delivering count. If this issue is happening due to networking trouble, then bridge should auto recover in some time. If new messages come on queue, they are getting processed but the older delivering count messages are never cleared on its own. If we stop and start the bridges then most of the time, these delivering messages are cleared. We are not able to find the root cause behind this incident. Any input/lead will be very helpful. Please also note that at the same time, we have other bridges b/w same source and target clusters on different addresses but we are not facing this challenge there. Source node1 <bridge name="FILES.RECORDER.1"> <queue-name>FILES.RECORDER</queue-name> <forwarding-address>FILES.RECORDER</forwarding-address> <retry-interval>45000</retry-interval> <max-retry-interval>45000</max-retry-interval> <reconnect-attempts>-1</reconnect-attempts> <static-connectors> <connector-ref>target-01</connector-ref> </static-connectors> </bridge> <bridge name="FILES.RECORDER.2"> <queue-name>FILES.RECORDER</queue-name> <forwarding-address>FILES.RECORDER</forwarding-address> <retry-interval>45000</retry-interval> <max-retry-interval>45000</max-retry-interval> <reconnect-attempts>-1</reconnect-attempts> <static-connectors> <connector-ref>target-02</connector-ref> </static-connectors> </bridge> Source node2 <bridge name="FILES.RECORDER.1"> <queue-name>FILES.RECORDER</queue-name> <forwarding-address>FILES.RECORDER</forwarding-address> <retry-interval>45000</retry-interval> <max-retry-interval>45000</max-retry-interval> <reconnect-attempts>-1</reconnect-attempts> <static-connectors> <connector-ref>target-01</connector-ref> </static-connectors> </bridge> <bridge name="FILES.RECORDER.2"> <queue-name>FILES.RECORDER</queue-name> <forwarding-address>FILES.RECORDER</forwarding-address> <retry-interval>45000</retry-interval> <max-retry-interval>45000</max-retry-interval> <reconnect-attempts>-1</reconnect-attempts> <static-connectors> <connector-ref>target-02</connector-ref> </static-connectors> </bridge>