Hi Tim,
first the respectve lines from karaf.log (the mobile broker is running in
Karaf). You can see the successful connection at 04:46:01,955. Then, bit
before 05:37:11, the mobile radio connection obviously was lost (I can also
see this from a process which monitors the ppp connection - reported a loss
of connectivity at ~ 05:36:40). Then there are several lines (some skipped
here since they always look the same) where re-connection of the broker
failed (INetAddress lookup failure, since the DNS was not reachable. Last
time at 05:37:27. Then, at 05:37:51,444, the mobile broker logs successful
connection.
2016-11-14 04:46:00,189 | INFO | ActiveMQ Task-16 |
DiscoveryNetworkConnector | 62 - org.apache.activemq.activemq-osgi -
5.14.1 | Establishing network connection from
vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9?async=false&create=false
to
failover:(ssl://backoffice1.abc.com:61617,ssl://backoffice2.abc.com:61617)?maxReconnectAttempts=0&randomize=true
2016-11-14 04:46:01,955 | INFO | ActiveMQ Task-1 | FailoverTransport
| 62 - org.apache.activemq.activemq-osgi - 5.14.1 | Successfully connected
to ssl://backoffice1.abc.com:61617
2016-11-14 04:46:02,072 | INFO | 9f6ce7f804a9#384 |
DemandForwardingBridgeSupport | 62 - org.apache.activemq.activemq-osgi -
5.14.1 | Network connection between
vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9#384 and
ssl://backoffice1.abc.com:61617 (Backoffice_Broker_1) has been established.
2016-11-14 05:37:11,992 | WARN | tyMonitor Worker | FailoverTransport
| 62 - org.apache.activemq.activemq-osgi - 5.14.1 | Transport
(ssl://backoffice1.abc.com:61617) failed , not attempting to automatically
reconnect: org.apache.activemq.transport.InactivityIOException: Channel was
inactive for too (>30000) long: tcp://aaa.bbb.ccc.ddd:61617
2016-11-14 05:37:11,995 | INFO | tyMonitor Worker |
DemandForwardingBridgeSupport | 62 - org.apache.activemq.activemq-osgi -
5.14.1 | Network connection between
vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9#384 and
unconnected shutdown due to a local error:
org.apache.activemq.transport.TransportDisposedIOException: Disposed due to
prior exception
2016-11-14 05:37:12,073 | INFO | 7f804a9] Task-69 |
DemandForwardingBridgeSupport | 62 - org.apache.activemq.activemq-osgi -
5.14.1 | MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9 bridge to
Backoffice_Broker_1 stopped
2016-11-14 05:37:13,006 | INFO | ActiveMQ Task-17 |
DiscoveryNetworkConnector | 62 - org.apache.activemq.activemq-osgi -
5.14.1 | Establishing network connection from
vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9?async=false&create=false
to
failover:(ssl://backoffice1.abc.com:61617,ssl://backoffice2.abc.com:61617)?maxReconnectAttempts=0&randomize=true
2016-11-14 05:37:13,044 | ERROR | ActiveMQ Task-17 | FailoverTransport
| 62 - org.apache.activemq.activemq-osgi - 5.14.1 | Failed to Lookup
INetAddress for URI[ssl://backoffice2.abc.com:61617] :
java.net.UnknownHostException: backoffice2.abc.com: unknown error
2016-11-14 05:37:13,278 | ERROR | ActiveMQ Task-1 | FailoverTransport
| 62 - org.apache.activemq.activemq-osgi - 5.14.1 | Failed to connect to
[ssl://backoffice1.abc.com:61617, ssl://backoffice2.abc.com:61617] after: 1
attempt(s)
2016-11-14 05:37:13,280 | WARN | 9f6ce7f804a9#388 |
DemandForwardingBridgeSupport | 62 - org.apache.activemq.activemq-osgi -
5.14.1 | Network connection between
vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9#388 and
unconnected shutdown due to a remote error:
java.util.concurrent.TimeoutException
2016-11-14 05:37:13,310 | INFO | 7f804a9] Task-69 |
DemandForwardingBridgeSupport | 62 - org.apache.activemq.activemq-osgi -
5.14.1 | MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9 bridge to
Unknown stopped
....
2016-11-14 05:37:27,525 | INFO | ActiveMQ Task-17 |
DiscoveryNetworkConnector | 62 - org.apache.activemq.activemq-osgi -
5.14.1 | Establishing network connection from
vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9?async=false&create=false
to
failover:(ssl://backoffice1.abc.com:61617,ssl://backoffice2.abc.com:61617)?maxReconnectAttempts=0&randomize=true
2016-11-14 05:37:27,534 | ERROR | ActiveMQ Task-17 | FailoverTransport
| 62 - org.apache.activemq.activemq-osgi - 5.14.1 | Failed to Lookup
INetAddress for URI[ssl://backoffice2.abc.com:61617] :
java.net.UnknownHostException: backoffice2.abc.com: unknown error
2016-11-14 05:37:27,674 | ERROR | ActiveMQ Task-1 | FailoverTransport
| 62 - org.apache.activemq.activemq-osgi - 5.14.1 | Failed to connect to
[ssl://backoffice1.abc.com:61617, ssl://backoffice2.abc.com:61617] after: 1
attempt(s)
2016-11-14 05:37:27,675 | WARN | 9f6ce7f804a9#400 |
DemandForwardingBridgeSupport | 62 - org.apache.activemq.activemq-osgi -
5.14.1 | Network connection between
vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9#400 and
unconnected shutdown due to a remote error:
java.util.concurrent.TimeoutException
2016-11-14 05:37:27,679 | INFO | 7f804a9] Task-72 |
DemandForwardingBridgeSupport | 62 - org.apache.activemq.activemq-osgi -
5.14.1 | MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9 bridge to
Unknown stopped
2016-11-14 05:37:43,678 | INFO | ActiveMQ Task-17 |
DiscoveryNetworkConnector | 62 - org.apache.activemq.activemq-osgi -
5.14.1 | Establishing network connection from
vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9?async=false&create=false
to
failover:(ssl://backoffice1.abc.com:61617,ssl://backoffice2.abc.com:61617)?maxReconnectAttempts=0&randomize=true
2016-11-14 05:37:51,444 | INFO | ActiveMQ Task-1 | FailoverTransport
| 62 - org.apache.activemq.activemq-osgi - 5.14.1 | Successfully connected
to ssl://backoffice1.abc.com:61617
Next activemq.log from the backoffice broker backoffice1 (there is also a
backoffice2, both are connected via a broker network in the backoffice - in
this example the successful connections were made between the mobile broker
and backoffice1). You again can see the successful connection at
04:46:01,397, and the loss of connection at 05:37:11,154. In between, there
are entries related to other mobile brokers (omitted here). After
05:37:11,154, there was no more entry in the log for the
MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9 in the backoffice
activemq.log, although the log of the mobile broker (karaf.log, see above)
shows a successful reconnection.
2016-11-14 04:46:01,390 | INFO | Started responder end of duplex bridge
94170c6d-3af8-4d7f-af53-9f6ce7f804a9_backoffice@ID:GERBVGTRAM1527-57289-1479027266276-0:1
| org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
ssl:///aaa.bbb.ccc.ddd:46403
2016-11-14 04:46:01,397 | INFO | Network connection between
vm://Backoffice_Broker_1#3036 and ssl:///aaa.bbb.ccc.ddd:46403
(MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9) has been
established. | org.apache.activemq.network.DemandForwardingBridgeSupport |
triggerStartAsyncNetworkBridgeCreation:
remoteBroker=ssl:///aaa.bbb.ccc.ddd:46403, localBroker=
vm://Backoffice_Broker_1#3036
...
2016-11-14 05:37:11,146 | WARN | Network connection between
vm://Backoffice_Broker_1#3036 and ssl:///aaa.bbb.ccc.ddd:46403 shutdown due
to a remote error: org.apache.activemq.transport.InactivityIOException:
Channel was inactive for too (>30000) long: tcp://aaa.bbb.ccc.ddd:46403 |
org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
InactivityMonitor Worker
2016-11-14 05:37:11,154 | INFO | Backoffice_Broker_1 bridge to
MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9 stopped |
org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
BrokerService[Backoffice_Broker_1] Task-6927
In the log of the second backoffice broker, there was no entry for this
mobile broker in activemq.log in that time frame. For completeness, here the
network connector definition on the mobile broker:
<networkConnectors>
<networkConnector
uri="static:(failover:(ssl://${backofficeBroker1}:61617,ssl://${backofficeBroker2}:61617)?maxReconnectAttempts=0&randomize=true)"
name="${deviceId}_backoffice"
dynamicOnly="true"
networkTTL="3"
duplex="true"
conduitSubscriptions="false"
decreaseNetworkConsumerPriority="false">
<dynamicallyIncludedDestinations>
<queue
physicalName="queue.platform.mobileToBackoffice.${queuePostfix}"/>
<queue physicalName="*.*.*.${deviceId}"/>
<topic
physicalName="topic.platform.backofficeToMobile.all"/>
</dynamicallyIncludedDestinations>
</networkConnector>
</networkConnectors>
On the backoffice side, there is a simple ssl connector. As already
mentioned, most of the times it works. Only sometimes, this behavior
appears. The bad thing is that the mobile broker looks like thinking that
everything is ok, and after this, it seems like it also doesn't get a signal
any longer that connectivity is lost, so it stays in this status. In the
Karaf console, activemq:dstat shows that no backoffice consumers are on the
queues.
Best Regards,
Jochen
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Network-of-brokers-consumers-not-synchronized-tp4718852p4719226.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
