Hello, I am working on a project that uses a network of ActiveMQ brokers. We use a fixed set of network connectors. The test team has been performing outage tests by shutting down services, killing services, and rebooting servers.
After the outage test, they give the system a moment to recover, then they check the consumer counts for the queues and topics. They found that sometimes, the consumer counts on a few queues are inconsistent between brokers. The client has connected to the broker without error (a local connection), and that broker shows a consumer. But the broker with a producer connection has a consumer count of 0. In this situation, the producer is still trying to send messages, but they never get to the consumer. Does anyone have advice on how to debug this? We've turned on debug, but its really noisy and we are not sure if there are specific messages or loggers that we should be looking for. We have seen some errors on other brokers when a rebooted system comes back and tries to connect again (this was when the broker wasn't shut down nicely). They other brokers seem to think they still have a connection to that broker (no TCP SYN sent from rebooted broker I assume?). I'm wondering, could it be that there is some sort of race condition when the original connection is cleaned up while a new one is created, and perhaps the new advertisement for consumer interest is cleaned up with it? A few things we have noticed that seem to contribute to the problem: * The AMQ service was not setup correctly to stop nicely on reboot, so it was likely just killed. When this was corrected, we had difficulty reproducing the problem. * It seems limited to queues the consumer creates manually using an ActiveMQConnectionFactory. The camel queues don't seem affected (sorry, not too familiar with camel yet). The ActiveMQConnectionFactory is created like this: Then a connection pool is created with it ("rawFactory" is the factory created above): This is a broker config from one of the servers. They are all the same, except for the tweak to the networkConnectors list. Any hints or tips would be greatly appreciated! Thank you, Frank -- View this message in context: http://activemq.2283324.n4.nabble.com/Zero-consumers-on-some-brokers-after-reboot-tp4713319.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.