RE: camel-rabbitmq clustering failover issue

[Valdis Andersons]

Never mind, we figured it out. Turns out mirroring is configured on the broker 
side since RabbitMQ 3.0 (:facepalm).
Once the primary went down Camel couldn't do anything with the in-flight 
messages anymore since the queues were not mirrored on the secondary vm. So the 
below is probably a very expected outcome of that.

Thanks and apologies for this,
Valdis

From: Valdis Andersons
Sent: 24 September 2018 15:16
To: users@camel.apache.org
Subject: camel-rabbitmq clustering failover issue

Hi All,

Hopefully someone can help me out here. In the project I'm working on we're 
using the camel-rabbitmq component to connect to a cluster of three RabbitMQ 
nodes. We're using the addresses parameter in the URL configuration of the 
RabbitMQ facing endpoints, like this:

from:rabbitmq://vmsys:5671/emailExchange?connectionFactory=rabbitConnectionFactory&autoDelete=false&queue=emailrouteQueue&exchangeType=topic&autoAck=false&addresses=vmsys:5671,vmsys2:5671,vmsys3:5671

Today the primary node 'vmsys' in that cluster got brought down for some OS 
patches and Camel in turn started logging this:

WARN  o.a.c.c.r.RabbitConsumer  - Unable to obtain a RabbitMQ channel. Will try 
again
INFO  o.a.c.c.r.RabbitConsumer  - Attempting to open a new rabbitMQ channel

Tried a restart of the application in the hopes that Camel might pick up the 
next node 'vmsys2' in the addresses list but no such thing, instead this was 
printed in the logs:

com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: 
#method<channel.close>(reply-code=404, reply-text=NOT_FOUND - home node 
'rabbit@vmsys' of durable queue 'emailrouteQueue' in vhost '/comms' is down or 
inaccessible, class-id=50, method-id=10)

In the RabbitMQ console on vmsys2 the queue statuses were showing up with a red 
flag and a message stating 'The queue is located on a cluster node or nodes 
that are down'.

I would have expected that Camel would simply switch over to the second node in 
the addresses list if the primary goes down. Also, while I'm seeing the queues 
from the vmsys2 node's console application it doesn't look like they're 
actually being replicated there like for our Spring application queues.
Is there something wrong in endpoint configuration that you can see or the 
connection factory setup? Think I followed the docs as best as I could.

Here is also how the rabbitConnectionFactory is instantiated:

@Bean(name = "rabbitConnectionFactory")
    public ConnectionFactory customerCommsCachingConnectionFactory() throws 
KeyManagementException, NoSuchAlgorithmException {
        ConnectionFactory cf = new ConnectionFactory();
        cf.setHost(rabbitHost);
       cf.setPort(rabbitPort);
        cf.setUsername(rabbitUser);
        cf.setPassword(rabbitPassword);
        cf.setVirtualHost(rabbitVirtualhost);
        cf.setAutomaticRecoveryEnabled(false);
        if (sslEnabled)
                cf.useSslProtocol();
        return cf;
    }

Any help on this would be greatly appreciated. If you need any more info on 
this please let me know, I'll do my best to supply it.


Thanks and Regards,
Valdis