Unsubscribe On Jun 24, 2016 10:16 AM, "FrankS" <doheth+activ...@gmail.com> wrote:
> Hello, > > I am working on a project that uses a network of ActiveMQ brokers. We use > a > fixed set of network connectors. The test team has been performing outage > tests by shutting down services, killing services, and rebooting servers. > > After the outage test, they give the system a moment to recover, then they > check the consumer counts for the queues and topics. They found that > sometimes, the consumer counts on a few queues are inconsistent between > brokers. The client has connected to the broker without error (a local > connection), and that broker shows a consumer. But the broker with a > producer connection has a consumer count of 0. In this situation, the > producer is still trying to send messages, but they never get to the > consumer. > > Does anyone have advice on how to debug this? We've turned on debug, but > its really noisy and we are not sure if there are specific messages or > loggers that we should be looking for. > > We have seen some errors on other brokers when a rebooted system comes back > and tries to connect again (this was when the broker wasn't shut down > nicely). They other brokers seem to think they still have a connection to > that broker (no TCP SYN sent from rebooted broker I assume?). I'm > wondering, could it be that there is some sort of race condition when the > original connection is cleaned up while a new one is created, and perhaps > the new advertisement for consumer interest is cleaned up with it? > > A few things we have noticed that seem to contribute to the problem: > * The AMQ service was not setup correctly to stop nicely on reboot, so it > was likely just killed. When this was corrected, we had difficulty > reproducing the problem. > * It seems limited to queues the consumer creates manually using an > ActiveMQConnectionFactory. The camel queues don't seem affected (sorry, > not > too familiar with camel yet). > > > The ActiveMQConnectionFactory is created like this: > > > > Then a connection pool is created with it ("rawFactory" is the factory > created above): > > > > > This is a broker config from one of the servers. They are all the same, > except for the tweak to the networkConnectors list. > > > > > Any hints or tips would be greatly appreciated! > > Thank you, > Frank > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/Zero-consumers-on-some-brokers-after-reboot-tp4713319.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >