Hi Have you been able to try with a newer release such as Camel 2.23.0, as the camel-rabbitmq component etc may been improved / bug fixed since that version you are using.
On Mon, Nov 12, 2018 at 5:34 PM Valdis Andersons <valdis.anders...@vhi.ie> wrote: > > Hi All, > > > > Hoping someone with more experience with RabbitMQ and Camel might be able to > tell me what’s going on with the below scenario as I’m not sure I understand > it right. > > > > We have now 3 servers in a test environment with RabbitMQ installed (v3.6.12, > Elang 20.1), we’re using camel-rabbitmq component (2.21.3) in our Camel > application to talk to the rabbit servers. Two of the servers (vm1 and vm2) > are in a cluster and have mirrored queues, the third server (vm3) is not in > the cluster but has federated queues and sits in a different data centre. > What we’re trying to figure out now is how to do disaster recovery flip-over > and it’s not working out too well so far. > > > > Here is an example of one of the consumer endpoints we have: > > > > rabbitmq://vm1:5671/someExchange?connectionFactory=rabbitConnectionFactory&autoDelete=false&queue=someQueue&exchangeType=topic&autoAck=false&bridgeEndpoint=true&concurrentConsumers=10&threadPoolSize=10&channelPoolMaxSize=10&prefetchCount=1&prefetchEnabled=true&exchangePattern=InOnly&automaticRecoveryEnabled=true&networkRecoveryInterval=15000&addresses=vm1:5671,vm2:5671,vm3:5671 > > > > Here is the connection factory (sslEnabled property is set to true): > > > > public ConnectionFactory rabbitConnectionFactory() throws > KeyManagementException, NoSuchAlgorithmException { > > ConnectionFactory cf = new ConnectionFactory(); > > cf.setHost(rabbitHost); > > cf.setPort(rabbitPort); > > cf.setUsername(rabbitUser); > > cf.setPassword(rabbitPassword); > > cf.setVirtualHost(rabbitVirtualhost); > > if (sslEnabled) > > cf.useSslProtocol(); > > return cf; > > } > > > > With only vm1 or vm2 going down it seems to work out just fine, the broker > seems to take care of the mirroring and the client side (Camel) rarely > notices a thing. Issues start when both of the cluster nodes (vm1 and vm2) go > down, for some reason the third address in the list isn’t being tried for the > recovery attempts. Without a Camel restart I’ve never seen the consumers > flipping over to vm3. > > > > In the logs I’m getting the following, which is most likely as expected: > > > > 2018-11-12 10:55:10,161 [Camel (camel-1) thread #13 - RabbitMQConsumer] INFO > o.a.c.c.r.RabbitConsumer - Received shutdown signal on the rabbitMQ channel > > 2018-11-12 10:55:10,161 [Camel (camel-1) thread #13 - RabbitMQConsumer] INFO > o.a.c.c.r.RabbitConsumer - Attempting to open a new rabbitMQ channel > > 2018-11-12 10:55:10,161 [AMQP Connection 10.1.1.1:5671] ERROR > c.r.c.i.ForgivingExceptionHandler - An unexpected connection driver error > occured > > javax.net.ssl.SSLException: Connection has been shutdown: > javax.net.ssl.SSLException: java.net.SocketException: Connection reset by > peer: socket write error > > at sun.security.ssl.SSLSocketImpl.checkEOF(Unknown Source) > > at sun.security.ssl.AppInputStream.read(Unknown Source) > > at java.io.BufferedInputStream.fill(Unknown Source) > > at java.io.BufferedInputStream.read(Unknown Source) > > at java.io.DataInputStream.readUnsignedByte(Unknown Source) > > at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:91) > > at > com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:164) > > at > com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:596) > > at java.lang.Thread.run(Unknown Source) > > Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection > reset by peer: socket write error > > at sun.security.ssl.Alerts.getSSLException(Unknown Source) > > at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source) > > at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source) > > at sun.security.ssl.SSLSocketImpl.handleException(Unknown > Source) > > at sun.security.ssl.SSLSocketImpl.handleException(Unknown > Source) > > at sun.security.ssl.AppOutputStream.write(Unknown Source) > > at java.io.BufferedOutputStream.flushBuffer(Unknown Source) > > at java.io.BufferedOutputStream.flush(Unknown Source) > > at java.io.DataOutputStream.flush(Unknown Source) > > at > com.rabbitmq.client.impl.SocketFrameHandler.flush(SocketFrameHandler.java:177) > > at > com.rabbitmq.client.impl.AMQConnection.flush(AMQConnection.java:571) > > at > com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:134) > > at > com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:447) > > at > com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:429) > > at > com.rabbitmq.client.impl.AMQConnection.handleConnectionClose(AMQConnection.java:846) > > at > com.rabbitmq.client.impl.AMQConnection.processControlCommand(AMQConnection.java:799) > > at > com.rabbitmq.client.impl.AMQConnection$1.processAsync(AMQConnection.java:242) > > at > com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:178) > > at > com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:111) > > at > com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:650) > > at > com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:48) > > at > com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:597) > > ... 1 common frames omitted > > Caused by: java.net.SocketException: Connection reset by peer: socket write > error > > at java.net.SocketOutputStream.socketWrite0(Native Method) > > at java.net.SocketOutputStream.socketWrite(Unknown Source) > > at java.net.SocketOutputStream.write(Unknown Source) > > at sun.security.ssl.OutputRecord.writeBuffer(Unknown Source) > > at sun.security.ssl.OutputRecord.write(Unknown Source) > > at sun.security.ssl.SSLSocketImpl.writeRecordInternal(Unknown > Source) > > at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source) > > ... 18 common frames omitted > > > > Then it starts the recovery attempts: > > > > 2018-11-12 13:47:57,846 [Camel (camel-1) thread #103 - RabbitMQConsumer] WARN > o.a.c.c.r.RabbitConsumer - Unable to obtain a RabbitMQ channel. Will try > again. Caused by: Waiting for channel to re-open.. Stacktrace logged at DEBUG > logging level. > > 2018-11-12 13:47:57,861 [Camel (camel-1) thread #19 - RabbitMQConsumer] WARN > o.a.c.c.r.RabbitConsumer - Unable to obtain a RabbitMQ channel. Will try > again. Caused by: Waiting for channel to re-open.. Stacktrace logged at DEBUG > logging level. > > 2018-11-12 13:47:57,861 [Camel (camel-1) thread #28 - RabbitMQConsumer] WARN > o.a.c.c.r.RabbitConsumer - Unable to obtain a RabbitMQ channel. Will try > again. Caused by: Waiting for channel to re-open.. Stacktrace logged at DEBUG > logging level. > > > > It prints that for each consumer/channel I guess, so there is a lot of those > lines, but for some reason the last available address in the addresses list > (vm3) is not being attempted to connect to (needs a restart to do that, so > defeats the purpose of automated recovery). > > Also, when the primary two nodes are brought back up the recovery is still > not recovering, I waited for about 30min and it still didn’t recover, just > kept printing the above warnings. > > When I tried to stop our app server (Tomcat) that hosts the Camel application > the above warnings started to be printed in their thousands and I got about > 500MB of logs inside a minute, all with the same warning message as above > (guess the retry timeout got ignored on shutdown while still trying to > recover the connections). > > > > I’d highly appreciate some insights into the above as I’m a bit lost on how > this is actually expected to work, I’ve been reading both RabbitMQ and Camel > docs but I’m none the wiser of how to do the DR flip-over or if that’s even > possible to do. I really hope it’s just some trivial config thing we’ve been > missing so far. > > > > > > Thanks, > > Valdis > > Vhi Group DAC (Vhi) is a holding company for insurance and healthcare > services, which include Vhi Healthcare DAC, Vhi Insurance DAC, Vhi Health > Services DAC and Vhi Investments DAC. Vhi Healthcare DAC trading as Vhi > Healthcare and Vhi Insurance DAC trading as Vhi Insurance are regulated by > the Central Bank of Ireland. Vhi Healthcare is tied to Vhi Insurance DAC for > health insurance in Ireland which is underwritten by Vhi Insurance DAC. Vhi > Healthcare is tied to Zurich Life Assurance plc for Vhi Life Term Insurance > and Vhi Mortgage Protection which is underwritten by Zurich Life Assurance > plc. Vhi Healthcare is tied to Collinson Insurance Services Limited for > MultiTrip Travel Insurance, Backpacker Travel Insurance and Vhi Dental > Insurance which are underwritten by Great Lakes Insurance SE, UK branch and > for Vhi Canada Cover and Vhi International Health Insurance which are > underwritten by Astrenska Insurance Limited. For more information about the > Vhi Group please go to: https://www.vhi.ie/about-vhi. > > > Tá Vhi Group DAC (Vhi) ina chuideachta sealbhaíochta le haghaidh seirbhísí > árachais agus seirbhísí cúram sláinte, lena n-áirítear Vhi Healthcare DAC, > Vhi Insurance DAC, Vhi Health Services DAC agus Vhi Investments DAC. Déanann > Banc Ceannais na hÉireann rialáil ar Vhi Healthcare DAC, ag trádáil dó mar > Vhi Healthcare, agus ar Vhi Insurance DAC, ag trádáil dó mar Vhi Insurance. > Tá Vhi Healthcare ceangailte le Vhi Insurance DAC le haghaidh árachas sláinte > in Éirinn, rud atá frithgheallta ag Vhi Insurance DAC. Tá Vhi Healthcare > ceangailte le Zurich Life Assurance plc le haghaidh Árachais Saoil de chuid > Vhi agus Árachas Cosanta Morgáiste de chuid Vhi atá frithgheallta ag Zurich > Life Assurance plc. Tá Vhi Healthcare ceangailte le Collinson Insurance > Services Limited le haghaidh Árachas Taistil Ilturais agus Turasóirí Mála > Droma agus Árachas Fiaclóireachta de chuid Vhi atá frithgheallta ag Great > Lakes Insurance SE, UK branch agus le haghaidh Clúdach Cheanada de chuid Vhi > agus Árachas Sláinte Idirnáisiúnta de chuid Vhi atá frithgheallta ag > Astrenska Insurance Limited. Chun tuilleadh faisnéise a fháil faoi Ghrúpa > Vhi, tabhair cuairt ar: https://www.vhi.ie/about-vhi. > > This e-mail and any files transmitted with it contain information which may > be confidential and which may also be privileged and is intended solely for > the use of the individual or entity to whom it is addressed. Unless you are > the intended recipient you may not copy or use it, or disclose it to anyone > else. Any opinions expressed are that of the individual and not necessarily > that of the Vhi Group. If you have received this e-mail in error please > notify the sender by return. > > > > > > > -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2