*My Setup: * Systems: >OS: Windows Servers 2016 >ActiveMQ Version: 5.14.5 >Java: 1.8.141
I have setup 2 - 3 node Master/Slave/Slave clusters on AWS 2 on-prem ActiveMQ servers >1 running 5.14.5 >1 running 5.10.0 Current network configuration Germany (AWS) <-> USEast (AWS) <-> Cambridge (on-prem) *Problem:* I have been seeing my network connector drop and reconnect every 30 sec or so. *Network Connector Config:* *Fmigr (DEXXXq01)* <networkConnectors> <networkConnector name="fmiuseast1" duplex="true" networkTTL="2" staticBridge="true" userName="${bridge.username}" password="${bridge.password}" uri="masterslave:(ssl://USAEXXXQ01.corp.local:61617,ssl://USAEXXXQ02.corp.local:61617,ssl://USAEXXXQ03.corp.local:61617)"> <staticallyIncludedDestinations> <topic physicalName="trfTrackingEvent" /> </staticallyIncludedDestinations> </networkConnector> <networkConnector name="fmiuseast2" duplex="false" networkTTL="2" staticBridge="true" userName="${bridge.username}" password="${bridge.password}" uri="masterslave:(ssl://USAEXXXQ01.corp.local:61617,ssl://USAEXXXQ02.corp.local:61617,ssl://USAEXXXQ03.corp.local:61617)"> <staticallyIncludedDestinations> <topic physicalName="medicalreport2" /> <topic physicalName="VirtualTopic.heartbeat1" /> </staticallyIncludedDestinations> </networkConnector> </networkConnectors> <transportConnectors> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?connectionTimeout=0&keepAlive=true&useInactivityMonitor=false&wireFormat.maxInactivityDuration=0"/> <transportConnector name="amqp" uri="amqp://0.0.0.0:5672?connectionTimeout=0&keepAlive=true&useInactivityMonitor=false&wireFormat.maxInactivityDuration=0"/> <transportConnector name="stomp" uri="stomp://0.0.0.0:61613?connectionTimeout=0&keepAlive=true&useInactivityMonitor=false&wireFormat.maxInactivityDuration=0"/> <transportConnector name="ssl" uri="ssl://0.0.0.0:61617?transport.enabledProtocols=TLSv1,TLSv1.1,TLSv1.2&connectionTimeout=0&keepAlive=true&useInactivityMonitor=false&wireFormat.maxInactivityDuration=0"/> <transportConnector name="stomp+ssl" uri="stomp+ssl://0.0.0.0:61612?connectionTimeout=0&keepAlive=true&useInactivityMonitor=false&wireFormat.maxInactivityDuration=0"/> <transportConnector name="amqp+ssl" uri="amqp+ssl://0.0.0.0:5671?connectionTimeout=0&keepAlive=true&useInactivityMonitor=false&wireFormat.maxInactivityDuration=0"/> </transportConnectors> *USEast Side:* <networkConnectors> <networkConnector name="fmigr" duplex="false" networkTTL="2" staticBridge="true" userName="${bridge.username}" password="${bridge.password}" uri="masterslave:(ssl://DEXXXQ01:61617,ssl://DEXXXQ02:61617,ssl://DEXXXQ03:61617)"> <staticallyIncludedDestinations> <topic physicalName="medicalReportingNotification" /> <topic physicalName="VirtualTopic.heartbeat" /> </staticallyIncludedDestinations> </networkConnector> </networkConnectors> <transportConnectors> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?connectionTimeout=0&keepAlive=true&useInactivityMonitor=false&wireFormat.maxInactivityDuration=0"/> <transportConnector name="amqp" uri="amqp://0.0.0.0:5672?connectionTimeout=0&keepAlive=true&useInactivityMonitor=false&wireFormat.maxInactivityDuration=0"/> <transportConnector name="stomp" uri="stomp://0.0.0.0:61613?connectionTimeout=0&keepAlive=true&useInactivityMonitor=false&wireFormat.maxInactivityDuration=0"/> <transportConnector name="ssl" uri="ssl://0.0.0.0:61617?transport.enabledProtocols=TLSv1,TLSv1.1,TLSv1.2&connectionTimeout=0&keepAlive=true&useInactivityMonitor=false&wireFormat.maxInactivityDuration=0"/> <transportConnector name="stomp+ssl" uri="stomp+ssl://0.0.0.0:61612?connectionTimeout=0&keepAlive=true&useInactivityMonitor=false&wireFormat.maxInactivityDuration=0"/> <transportConnector name="amqp+ssl" uri="amqp+ssl://0.0.0.0:5671?connectionTimeout=0&keepAlive=true&useInactivityMonitor=false&wireFormat.maxInactivityDuration=0"/> </transportConnectors> *My fmigr activeMQ Log:* 2017-09-19 18:25:02,578 | INFO | Establishing network connection from vm://fmigrq01?async=false&create=false to failover:(ssl://USAEXXXQ01.corp.local:61617,ssl://USAEXXXQ02.corp.local:61617,ssl://USAEXXXQ03.corp.local:61617)?randomize=false&maxReconnectAttempts=0 | org.apache.activemq.network.DiscoveryNetworkConnector | Simple Discovery Agent-34 2017-09-19 18:25:03,213 | INFO | Successfully connected to ssl://USAEXXXQ01.corp.local:61617 | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Failover Worker: 322718397 2017-09-19 18:25:03,321 | INFO | Network connection between vm://fmigrq01#178 and ssl://USAEXXXQ01.corp.local:61617 (fmiuseastq01) has been established. | org.apache.activemq.network.DemandForwardingBridgeSupport | triggerStartAsyncNetworkBridgeCreation: remoteBroker=unconnected, localBroker= vm://fmigrq01#178 2017-09-19 18:25:33,433 | WARN | Transport (ssl://USAEXXXQ01.corp.local:61617) failed , not attempting to automatically reconnect: java.io.EOFException | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Transport: ssl://USAEXXXQ01.corp.local/10.120.237.233:61617 2017-09-19 18:25:33,433 | WARN | Network connection between vm://fmigrq01#178 and unconnected shutdown due to a remote error: java.io.EOFException | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: ssl://USAEXXXQ01.corp.local/10.120.237.233:61617 2017-09-19 18:25:33,439 | INFO | fmigrq01 bridge to fmiuseastq01 stopped | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[fmigrq01] Task-298 2017-09-19 18:25:34,435 | INFO | Establishing network connection from vm://fmigrq01?async=false&create=false to failover:(ssl://USAEXXXQ01.corp.local:61617,ssl://USAEXXXQ02.corp.local:61617,ssl://USAEXXXQ03.corp.local:61617)?randomize=false&maxReconnectAttempts=0 | org.apache.activemq.network.DiscoveryNetworkConnector | Simple Discovery Agent-35 2017-09-19 18:25:35,062 | INFO | Successfully connected to ssl://USAEXXXQ01.corp.local:61617 | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Failover Worker: 663286350 2017-09-19 18:25:35,169 | INFO | Network connection between vm://fmigrq01#182 and ssl://USAEXXXQ01.corp.local:61617 (fmiuseastq01) has been established. | org.apache.activemq.network.DemandForwardingBridgeSupport | triggerStartAsyncNetworkBridgeCreation: remoteBroker=unconnected, localBroker= vm://fmigrq01#182 2017-09-19 18:26:05,301 | WARN | Transport (ssl://USAEXXXQ01.corp.local:61617) failed , not attempting to automatically reconnect: java.io.EOFException | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Transport: ssl://USAEXXXQ01.corp.local/10.120.237.233:61617 2017-09-19 18:26:05,302 | WARN | Network connection between vm://fmigrq01#182 and unconnected shutdown due to a remote error: java.io.EOFException | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: ssl://USAEXXXQ01.corp.local/10.120.237.233:61617 2017-09-19 18:26:05,307 | INFO | fmigrq01 bridge to fmiuseastq01 stopped | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[fmigrq01] Task-307 2017-09-19 18:26:06,304 | INFO | Establishing network connection from vm://fmigrq01?async=false&create=false to failover:(ssl://USAEXXXQ01.corp.local:61617,ssl://USAEXXXQ02.corp.local:61617,ssl://USAEXXXQ03.corp.local:61617)?randomize=false&maxReconnectAttempts=0 | org.apache.activemq.network.DiscoveryNetworkConnector | Simple Discovery Agent-36 2017-09-19 18:26:06,937 | INFO | Successfully connected to ssl://USAEXXXQ01.corp.local:61617 | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Failover Worker: 1852967876 2017-09-19 18:26:07,041 | INFO | Network connection between vm://fmigrq01#188 and ssl://USAEXXXQ01.corp.local:61617 (fmiuseastq01) has been established. | org.apache.activemq.network.DemandForwardingBridgeSupport | triggerStartAsyncNetworkBridgeCreation: remoteBroker=unconnected, localBroker= vm://fmigrq01#188 2017-09-19 18:26:37,156 | WARN | Transport (ssl://USAEXXXQ01.corp.local:61617) failed , not attempting to automatically reconnect: java.io.EOFException | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Transport: ssl://USAEXXXQ01.corp.local/10.120.237.233:61617 2017-09-19 18:26:37,156 | WARN | Network connection between vm://fmigrq01#188 and unconnected shutdown due to a remote error: java.io.EOFException | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: ssl://USAEXXXQ01.corp.local/10.120.237.233:61617 2017-09-19 18:26:37,162 | INFO | fmigrq01 bridge to fmiuseastq01 stopped | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[fmigrq01] Task-314 2017-09-19 18:26:38,158 | INFO | Establishing network connection from vm://fmigrq01?async=false&create=false to failover:(ssl://USAEXXXQ01.corp.local:61617,ssl://USAEXXXQ02.corp.local:61617,ssl://USAEXXXQ03.corp.local:61617)?randomize=false&maxReconnectAttempts=0 | org.apache.activemq.network.DiscoveryNetworkConnector | Simple Discovery Agent-37 2017-09-19 18:26:38,784 | INFO | Successfully connected to ssl://USAEXXXQ01.corp.local:61617 | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Failover Worker: 1838742535 2017-09-19 18:26:38,893 | INFO | Network connection between vm://fmigrq01#192 and ssl://USAEXXXQ01.corp.local:61617 (fmiuseastq01) has been established. | org.apache.activemq.network.DemandForwardingBridgeSupport | triggerStartAsyncNetworkBridgeCreation: remoteBroker=unconnected, localBroker= vm://fmigrq01#192 2017-09-19 18:27:09,006 | WARN | Transport (ssl://USAEXXXQ01.corp.local:61617) failed , not attempting to automatically reconnect: java.io.EOFException | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Transport: ssl://USAEXXXQ01.corp.local/10.120.237.233:61617 2017-09-19 18:27:09,006 | WARN | Network connection between vm://fmigrq01#192 and unconnected shutdown due to a remote error: java.io.EOFException | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: ssl://USAEXXXQ01.corp.local/10.120.237.233:61617 2017-09-19 18:27:09,012 | INFO | fmigrq01 bridge to fmiuseastq01 stopped | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[fmigrq01] Task-323 2017-09-19 18:27:10,008 | INFO | Establishing network connection from vm://fmigrq01?async=false&create=false to failover:(ssl://USAEXXXQ01.corp.local:61617,ssl://USAEXXXQ02.corp.local:61617,ssl://USAEXXXQ03.corp.local:61617)?randomize=false&maxReconnectAttempts=0 | org.apache.activemq.network.DiscoveryNetworkConnector | Simple Discovery Agent-38 2017-09-19 18:27:10,640 | INFO | Successfully connected to ssl://USAEXXXQ01.corp.local:61617 | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Failover Worker: 1244983570 2017-09-19 18:27:10,747 | INFO | Network connection between vm://fmigrq01#198 and ssl://USAEXXXQ01.corp.local:61617 (fmiuseastq01) has been established. | org.apache.activemq.network.DemandForwardingBridgeSupport | triggerStartAsyncNetworkBridgeCreation: remoteBroker=unconnected, localBroker= vm://fmigrq01#198 2017-09-19 18:27:40,859 | WARN | Transport (ssl://USAEXXXQ01.corp.local:61617) failed , not attempting to automatically reconnect: java.io.EOFException | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Transport: ssl://USAEXXXQ01.corp.local/10.120.237.233:61617 2017-09-19 18:27:40,859 | WARN | Network connection between vm://fmigrq01#198 and unconnected shutdown due to a remote error: java.io.EOFException | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: ssl://USAEXXXQ01.corp.local/10.120.237.233:61617 2017-09-19 18:27:40,865 | INFO | fmigrq01 bridge to fmiuseastq01 stopped | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[fmigrq01] Task-330 2017-09-19 18:27:41,861 | INFO | Establishing network connection from vm://fmigrq01?async=false&create=false to failover:(ssl://USAEXXXQ01.corp.local:61617,ssl://USAEXXXQ02.corp.local:61617,ssl://USAEXXXQ03.corp.local:61617)?randomize=false&maxReconnectAttempts=0 | org.apache.activemq.network.DiscoveryNetworkConnector | Simple Discovery Agent-39 2017-09-19 18:27:42,497 | INFO | Successfully connected to ssl://USAEXXXQ01.corp.local:61617 | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Failover Worker: 1096056638 2017-09-19 18:27:42,608 | INFO | Network connection between vm://fmigrq01#202 and ssl://USAEXXXQ01.corp.local:61617 (fmiuseastq01) has been established. | org.apache.activemq.network.DemandForwardingBridgeSupport | triggerStartAsyncNetworkBridgeCreation: remoteBroker=unconnected, localBroker= vm://fmigrq01#202 2017-09-19 18:28:12,717 | WARN | Transport (ssl://USAEXXXQ01.corp.local:61617) failed , not attempting to automatically reconnect: java.io.EOFException | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Transport: ssl://USAEXXXQ01.corp.local/10.120.237.233:61617 2017-09-19 18:28:12,717 | WARN | Network connection between vm://fmigrq01#202 and unconnected shutdown due to a remote error: java.io.EOFException | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: ssl://USAEXXXQ01.corp.local/10.120.237.233:61617 2017-09-19 18:28:12,724 | INFO | fmigrq01 bridge to fmiuseastq01 stopped | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[fmigrq01] Task-339 2017-09-19 18:28:13,719 | INFO | Establishing network connection from vm://fmigrq01?async=false&create=false to failover:(ssl://USAEXXXQ01.corp.local:61617,ssl://USAEXXXQ02.corp.local:61617,ssl://USAEXXXQ03.corp.local:61617)?randomize=false&maxReconnectAttempts=0 | org.apache.activemq.network.DiscoveryNetworkConnector | Simple Discovery Agent-40 2017-09-19 18:28:14,349 | INFO | Successfully connected to ssl://USAEXXXQ01.corp.local:61617 | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Failover Worker: 597416418 2017-09-19 18:28:14,455 | INFO | Network connection between vm://fmigrq01#208 and ssl://USAEXXXQ01.corp.local:61617 (fmiuseastq01) has been established. | org.apache.activemq.network.DemandForwardingBridgeSupport | triggerStartAsyncNetworkBridgeCreation: remoteBroker=unconnected, localBroker= vm://fmigrq01#208 -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html