Hello,
I am working on a project that uses a network of ActiveMQ brokers. We use a
fixed set of network connectors. The test team has been performing outage
tests by shutting down services, killing services, and rebooting servers.
After the outage test, they give the system a moment to recover, then they
check the consumer counts for the queues and topics. They found that
sometimes, the consumer counts on a few queues are inconsistent between
brokers. The client has connected to the broker without error (a local
connection), and that broker shows a consumer. But the broker with a
producer connection has a consumer count of 0. In this situation, the
producer is still trying to send messages, but they never get to the
consumer.
Does anyone have advice on how to debug this? We've turned on debug, but
its really noisy and we are not sure if there are specific messages or
loggers that we should be looking for.
We have seen some errors on other brokers when a rebooted system comes back
and tries to connect again (this was when the broker wasn't shut down
nicely). They other brokers seem to think they still have a connection to
that broker (no TCP SYN sent from rebooted broker I assume?). I'm
wondering, could it be that there is some sort of race condition when the
original connection is cleaned up while a new one is created, and perhaps
the new advertisement for consumer interest is cleaned up with it?
A few things we have noticed that seem to contribute to the problem:
* The AMQ service was not setup correctly to stop nicely on reboot, so it
was likely just killed. When this was corrected, we had difficulty
reproducing the problem.
* It seems limited to queues the consumer creates manually using an
ActiveMQConnectionFactory. The camel queues don't seem affected (sorry, not
too familiar with camel yet).
The ActiveMQConnectionFactory is created like this:
Then a connection pool is created with it ("rawFactory" is the factory
created above):
This is a broker config from one of the servers. They are all the same,
except for the tweak to the networkConnectors list.
Any hints or tips would be greatly appreciated!
Thank you,
Frank
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Zero-consumers-on-some-brokers-after-reboot-tp4713319.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
