When the problem occurs, do you see an outbound network connection from the
mobile broker to the backoffice broker? Does it go to the right host (i.e.
backoffice1) at the right IP address? And do you see a corresponding
inbound connection on the backoffice1 broker?
Also, is it expected that the mobile broker would be unable to resolve the
hostname of the backoffice broker during a period of disconnection? That
might be expected or unexpected depending on your network configuration, so
it might be a red herring, but I want to make sure we don't overlook it if
it's actually a problem.
Tim
On Nov 15, 2016 7:01 AM, "jochenw" <jochen.walz.m...@googlemail.com> wrote:
> Hi Tim,
>
> first the respectve lines from karaf.log (the mobile broker is running in
> Karaf). You can see the successful connection at 04:46:01,955. Then, bit
> before 05:37:11, the mobile radio connection obviously was lost (I can also
> see this from a process which monitors the ppp connection - reported a loss
> of connectivity at ~ 05:36:40). Then there are several lines (some skipped
> here since they always look the same) where re-connection of the broker
> failed (INetAddress lookup failure, since the DNS was not reachable. Last
> time at 05:37:27. Then, at 05:37:51,444, the mobile broker logs successful
> connection.
>
> 2016-11-14 04:46:00,189 | INFO | ActiveMQ Task-16 |
> DiscoveryNetworkConnector | 62 - org.apache.activemq.activemq-osgi
> -
> 5.14.1 | Establishing network connection from
> vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-
> 9f6ce7f804a9?async=false&create=false
> to
> failover:(ssl://backoffice1.abc.com:61617,ssl://backoffice2.abc.com:61617
> )?maxReconnectAttempts=0&randomize=true
> 2016-11-14 04:46:01,955 | INFO | ActiveMQ Task-1 | FailoverTransport
> | 62 - org.apache.activemq.activemq-osgi - 5.14.1 | Successfully connected
> to ssl://backoffice1.abc.com:61617
> 2016-11-14 04:46:02,072 | INFO | 9f6ce7f804a9#384 |
> DemandForwardingBridgeSupport | 62 - org.apache.activemq.activemq-osgi
> -
> 5.14.1 | Network connection between
> vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9#384 and
> ssl://backoffice1.abc.com:61617 (Backoffice_Broker_1) has been
> established.
> 2016-11-14 05:37:11,992 | WARN | tyMonitor Worker | FailoverTransport
> | 62 - org.apache.activemq.activemq-osgi - 5.14.1 | Transport
> (ssl://backoffice1.abc.com:61617) failed , not attempting to automatically
> reconnect: org.apache.activemq.transport.InactivityIOException: Channel
> was
> inactive for too (>30000) long: tcp://aaa.bbb.ccc.ddd:61617
> 2016-11-14 05:37:11,995 | INFO | tyMonitor Worker |
> DemandForwardingBridgeSupport | 62 - org.apache.activemq.activemq-osgi
> -
> 5.14.1 | Network connection between
> vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9#384 and
> unconnected shutdown due to a local error:
> org.apache.activemq.transport.TransportDisposedIOException: Disposed due
> to
> prior exception
> 2016-11-14 05:37:12,073 | INFO | 7f804a9] Task-69 |
> DemandForwardingBridgeSupport | 62 - org.apache.activemq.activemq-osgi
> -
> 5.14.1 | MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9 bridge
> to
> Backoffice_Broker_1 stopped
> 2016-11-14 05:37:13,006 | INFO | ActiveMQ Task-17 |
> DiscoveryNetworkConnector | 62 - org.apache.activemq.activemq-osgi
> -
> 5.14.1 | Establishing network connection from
> vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-
> 9f6ce7f804a9?async=false&create=false
> to
> failover:(ssl://backoffice1.abc.com:61617,ssl://backoffice2.abc.com:61617
> )?maxReconnectAttempts=0&randomize=true
> 2016-11-14 05:37:13,044 | ERROR | ActiveMQ Task-17 | FailoverTransport
> | 62 - org.apache.activemq.activemq-osgi - 5.14.1 | Failed to Lookup
> INetAddress for URI[ssl://backoffice2.abc.com:61617] :
> java.net.UnknownHostException: backoffice2.abc.com: unknown error
> 2016-11-14 05:37:13,278 | ERROR | ActiveMQ Task-1 | FailoverTransport
> | 62 - org.apache.activemq.activemq-osgi - 5.14.1 | Failed to connect to
> [ssl://backoffice1.abc.com:61617, ssl://backoffice2.abc.com:61617] after:
> 1
> attempt(s)
> 2016-11-14 05:37:13,280 | WARN | 9f6ce7f804a9#388 |
> DemandForwardingBridgeSupport | 62 - org.apache.activemq.activemq-osgi
> -
> 5.14.1 | Network connection between
> vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9#388 and
> unconnected shutdown due to a remote error:
> java.util.concurrent.TimeoutException
> 2016-11-14 05:37:13,310 | INFO | 7f804a9] Task-69 |
> DemandForwardingBridgeSupport | 62 - org.apache.activemq.activemq-osgi
> -
> 5.14.1 | MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9 bridge
> to
> Unknown stopped
> ....
> 2016-11-14 05:37:27,525 | INFO | ActiveMQ Task-17 |
> DiscoveryNetworkConnector | 62 - org.apache.activemq.activemq-osgi
> -
> 5.14.1 | Establishing network connection from
> vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-
> 9f6ce7f804a9?async=false&create=false
> to
> failover:(ssl://backoffice1.abc.com:61617,ssl://backoffice2.abc.com:61617
> )?maxReconnectAttempts=0&randomize=true
> 2016-11-14 05:37:27,534 | ERROR | ActiveMQ Task-17 | FailoverTransport
> | 62 - org.apache.activemq.activemq-osgi - 5.14.1 | Failed to Lookup
> INetAddress for URI[ssl://backoffice2.abc.com:61617] :
> java.net.UnknownHostException: backoffice2.abc.com: unknown error
> 2016-11-14 05:37:27,674 | ERROR | ActiveMQ Task-1 | FailoverTransport
> | 62 - org.apache.activemq.activemq-osgi - 5.14.1 | Failed to connect to
> [ssl://backoffice1.abc.com:61617, ssl://backoffice2.abc.com:61617] after:
> 1
> attempt(s)
> 2016-11-14 05:37:27,675 | WARN | 9f6ce7f804a9#400 |
> DemandForwardingBridgeSupport | 62 - org.apache.activemq.activemq-osgi
> -
> 5.14.1 | Network connection between
> vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9#400 and
> unconnected shutdown due to a remote error:
> java.util.concurrent.TimeoutException
> 2016-11-14 05:37:27,679 | INFO | 7f804a9] Task-72 |
> DemandForwardingBridgeSupport | 62 - org.apache.activemq.activemq-osgi
> -
> 5.14.1 | MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9 bridge
> to
> Unknown stopped
> 2016-11-14 05:37:43,678 | INFO | ActiveMQ Task-17 |
> DiscoveryNetworkConnector | 62 - org.apache.activemq.activemq-osgi
> -
> 5.14.1 | Establishing network connection from
> vm://MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-
> 9f6ce7f804a9?async=false&create=false
> to
> failover:(ssl://backoffice1.abc.com:61617,ssl://backoffice2.abc.com:61617
> )?maxReconnectAttempts=0&randomize=true
> 2016-11-14 05:37:51,444 | INFO | ActiveMQ Task-1 | FailoverTransport
> | 62 - org.apache.activemq.activemq-osgi - 5.14.1 | Successfully connected
> to ssl://backoffice1.abc.com:61617
>
>
> Next activemq.log from the backoffice broker backoffice1 (there is also a
> backoffice2, both are connected via a broker network in the backoffice - in
> this example the successful connections were made between the mobile broker
> and backoffice1). You again can see the successful connection at
> 04:46:01,397, and the loss of connection at 05:37:11,154. In between, there
> are entries related to other mobile brokers (omitted here). After
> 05:37:11,154, there was no more entry in the log for the
> MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9 in the
> backoffice
> activemq.log, although the log of the mobile broker (karaf.log, see above)
> shows a successful reconnection.
>
> 2016-11-14 04:46:01,390 | INFO | Started responder end of duplex bridge
> 94170c6d-3af8-4d7f-af53-9f6ce7f804a9_backoffice@ID:GERBVGTRAM1527-57289-
> 1479027266276-0:1
> | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> ssl:///aaa.bbb.ccc.ddd:46403
> 2016-11-14 04:46:01,397 | INFO | Network connection between
> vm://Backoffice_Broker_1#3036 and ssl:///aaa.bbb.ccc.ddd:46403
> (MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9) has been
> established. | org.apache.activemq.network.DemandForwardingBridgeSupport |
> triggerStartAsyncNetworkBridgeCreation:
> remoteBroker=ssl:///aaa.bbb.ccc.ddd:46403, localBroker=
> vm://Backoffice_Broker_1#3036
> ...
> 2016-11-14 05:37:11,146 | WARN | Network connection between
> vm://Backoffice_Broker_1#3036 and ssl:///aaa.bbb.ccc.ddd:46403 shutdown due
> to a remote error: org.apache.activemq.transport.InactivityIOException:
> Channel was inactive for too (>30000) long: tcp://aaa.bbb.ccc.ddd:46403 |
> org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
> InactivityMonitor Worker
> 2016-11-14 05:37:11,154 | INFO | Backoffice_Broker_1 bridge to
> MOBILE_MessageBroker_94170c6d-3af8-4d7f-af53-9f6ce7f804a9 stopped |
> org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
> BrokerService[Backoffice_Broker_1] Task-6927
>
> In the log of the second backoffice broker, there was no entry for this
> mobile broker in activemq.log in that time frame. For completeness, here
> the
> network connector definition on the mobile broker:
>
> <networkConnectors>
> <networkConnector
> uri="static:(failover:(ssl://${backofficeBroker1}:61617,ssl:
> //${backofficeBroker2}:61617)?maxReconnectAttempts=0&randomize=true)"
> name="${deviceId}_backoffice"
> dynamicOnly="true"
> networkTTL="3"
> duplex="true"
> conduitSubscriptions="false"
> decreaseNetworkConsumerPriority="false">
> <dynamicallyIncludedDestinations>
> <queue
> physicalName="queue.platform.mobileToBackoffice.${queuePostfix}"/>
> <queue physicalName="*.*.*.${deviceId}"/>
> <topic
> physicalName="topic.platform.backofficeToMobile.all"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
> </networkConnectors>
>
> On the backoffice side, there is a simple ssl connector. As already
> mentioned, most of the times it works. Only sometimes, this behavior
> appears. The bad thing is that the mobile broker looks like thinking that
> everything is ok, and after this, it seems like it also doesn't get a
> signal
> any longer that connectivity is lost, so it stays in this status. In the
> Karaf console, activemq:dstat shows that no backoffice consumers are on the
> queues.
>
> Best Regards,
> Jochen
>
>
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Network-of-brokers-consumers-not-synchronized-
> tp4718852p4719226.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
