Hi Claus, Thanks for the response, had forgotten about this email completely. In the meantime manged to figure out a working configuration - it's just leaving all the recovery settings as default. Adding the URL parameters "automaticRecoveryEnabled=true&networkRecoveryInterval=15000" just screws things up as outlined below. With just the recovery enabled on the connection factory (it's on by default) it works correctly. I should have probably tested the clustering before adding any new URL properties (like a baseline test), that would have revealed that recovery is on and working by default. Live and learn I guess. Maybe a note could be added to the documentation stating that those parameters are not needed on later versions of Camel?
Thanks, Valdis -----Original Message----- From: Claus Ibsen [mailto:claus.ib...@gmail.com] Sent: 03 January 2019 12:18 To: users@camel.apache.org Subject: Re: camel-rabbitmq automated recovery Hi Have you been able to try with a newer release such as Camel 2.23.0, as the camel-rabbitmq component etc may been improved / bug fixed since that version you are using. On Mon, Nov 12, 2018 at 5:34 PM Valdis Andersons <valdis.anders...@vhi.ie> wrote: > > Hi All, > > > > Hoping someone with more experience with RabbitMQ and Camel might be able to > tell me what’s going on with the below scenario as I’m not sure I understand > it right. > > > > We have now 3 servers in a test environment with RabbitMQ installed (v3.6.12, > Elang 20.1), we’re using camel-rabbitmq component (2.21.3) in our Camel > application to talk to the rabbit servers. Two of the servers (vm1 and vm2) > are in a cluster and have mirrored queues, the third server (vm3) is not in > the cluster but has federated queues and sits in a different data centre. > What we’re trying to figure out now is how to do disaster recovery flip-over > and it’s not working out too well so far. > > > > Here is an example of one of the consumer endpoints we have: > > > > rabbitmq://vm1:5671/someExchange?connectionFactory=rabbitConnectionFac > tory&autoDelete=false&queue=someQueue&exchangeType=topic&autoAck=false > &bridgeEndpoint=true&concurrentConsumers=10&threadPoolSize=10&channelP > oolMaxSize=10&prefetchCount=1&prefetchEnabled=true&exchangePattern=InO > nly&automaticRecoveryEnabled=true&networkRecoveryInterval=15000&addres > ses=vm1:5671,vm2:5671,vm3:5671 > > > > Here is the connection factory (sslEnabled property is set to true): > > > > public ConnectionFactory rabbitConnectionFactory() throws > KeyManagementException, NoSuchAlgorithmException { > > ConnectionFactory cf = new ConnectionFactory(); > > cf.setHost(rabbitHost); > > cf.setPort(rabbitPort); > > cf.setUsername(rabbitUser); > > cf.setPassword(rabbitPassword); > > cf.setVirtualHost(rabbitVirtualhost); > > if (sslEnabled) > > cf.useSslProtocol(); > > return cf; > > } > > > > With only vm1 or vm2 going down it seems to work out just fine, the broker > seems to take care of the mirroring and the client side (Camel) rarely > notices a thing. Issues start when both of the cluster nodes (vm1 and vm2) go > down, for some reason the third address in the list isn’t being tried for the > recovery attempts. Without a Camel restart I’ve never seen the consumers > flipping over to vm3. > > > > In the logs I’m getting the following, which is most likely as expected: > > > > 2018-11-12 10:55:10,161 [Camel (camel-1) thread #13 - > RabbitMQConsumer] INFO o.a.c.c.r.RabbitConsumer - Received shutdown > signal on the rabbitMQ channel > > 2018-11-12 10:55:10,161 [Camel (camel-1) thread #13 - > RabbitMQConsumer] INFO o.a.c.c.r.RabbitConsumer - Attempting to open > a new rabbitMQ channel > > 2018-11-12 10:55:10,161 [AMQP Connection 10.1.1.1:5671] ERROR > c.r.c.i.ForgivingExceptionHandler - An unexpected connection driver > error occured > > javax.net.ssl.SSLException: Connection has been shutdown: > javax.net.ssl.SSLException: java.net.SocketException: Connection reset > by peer: socket write error > > at sun.security.ssl.SSLSocketImpl.checkEOF(Unknown > Source) > > at sun.security.ssl.AppInputStream.read(Unknown > Source) > > at java.io.BufferedInputStream.fill(Unknown Source) > > at java.io.BufferedInputStream.read(Unknown Source) > > at java.io.DataInputStream.readUnsignedByte(Unknown > Source) > > at > com.rabbitmq.client.impl.Frame.readFrom(Frame.java:91) > > at > com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandl > er.java:164) > > at > com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java > :596) > > at java.lang.Thread.run(Unknown Source) > > Caused by: javax.net.ssl.SSLException: java.net.SocketException: > Connection reset by peer: socket write error > > at sun.security.ssl.Alerts.getSSLException(Unknown > Source) > > at sun.security.ssl.SSLSocketImpl.fatal(Unknown > Source) > > at sun.security.ssl.SSLSocketImpl.fatal(Unknown > Source) > > at > sun.security.ssl.SSLSocketImpl.handleException(Unknown Source) > > at > sun.security.ssl.SSLSocketImpl.handleException(Unknown Source) > > at sun.security.ssl.AppOutputStream.write(Unknown > Source) > > at java.io.BufferedOutputStream.flushBuffer(Unknown > Source) > > at java.io.BufferedOutputStream.flush(Unknown Source) > > at java.io.DataOutputStream.flush(Unknown Source) > > at > com.rabbitmq.client.impl.SocketFrameHandler.flush(SocketFrameHandler.j > ava:177) > > at > com.rabbitmq.client.impl.AMQConnection.flush(AMQConnection.java:571) > > at > com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:134) > > at > com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java: > 447) > > at > com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java: > 429) > > at > com.rabbitmq.client.impl.AMQConnection.handleConnectionClose(AMQConnec > tion.java:846) > > at > com.rabbitmq.client.impl.AMQConnection.processControlCommand(AMQConnec > tion.java:799) > > at > com.rabbitmq.client.impl.AMQConnection$1.processAsync(AMQConnection.ja > va:242) > > at > com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQCh > annel.java:178) > > at > com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:111) > > at > com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:65 > 0) > > at > com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:4 > 8) > > at > com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java > :597) > > ... 1 common frames omitted > > Caused by: java.net.SocketException: Connection reset by peer: socket > write error > > at java.net.SocketOutputStream.socketWrite0(Native > Method) > > at java.net.SocketOutputStream.socketWrite(Unknown > Source) > > at java.net.SocketOutputStream.write(Unknown Source) > > at sun.security.ssl.OutputRecord.writeBuffer(Unknown > Source) > > at sun.security.ssl.OutputRecord.write(Unknown Source) > > at > sun.security.ssl.SSLSocketImpl.writeRecordInternal(Unknown Source) > > at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown > Source) > > ... 18 common frames omitted > > > > Then it starts the recovery attempts: > > > > 2018-11-12 13:47:57,846 [Camel (camel-1) thread #103 - RabbitMQConsumer] WARN > o.a.c.c.r.RabbitConsumer - Unable to obtain a RabbitMQ channel. Will try > again. Caused by: Waiting for channel to re-open.. Stacktrace logged at DEBUG > logging level. > > 2018-11-12 13:47:57,861 [Camel (camel-1) thread #19 - RabbitMQConsumer] WARN > o.a.c.c.r.RabbitConsumer - Unable to obtain a RabbitMQ channel. Will try > again. Caused by: Waiting for channel to re-open.. Stacktrace logged at DEBUG > logging level. > > 2018-11-12 13:47:57,861 [Camel (camel-1) thread #28 - RabbitMQConsumer] WARN > o.a.c.c.r.RabbitConsumer - Unable to obtain a RabbitMQ channel. Will try > again. Caused by: Waiting for channel to re-open.. Stacktrace logged at DEBUG > logging level. > > > > It prints that for each consumer/channel I guess, so there is a lot of those > lines, but for some reason the last available address in the addresses list > (vm3) is not being attempted to connect to (needs a restart to do that, so > defeats the purpose of automated recovery). > > Also, when the primary two nodes are brought back up the recovery is still > not recovering, I waited for about 30min and it still didn’t recover, just > kept printing the above warnings. > > When I tried to stop our app server (Tomcat) that hosts the Camel application > the above warnings started to be printed in their thousands and I got about > 500MB of logs inside a minute, all with the same warning message as above > (guess the retry timeout got ignored on shutdown while still trying to > recover the connections). > > > > I’d highly appreciate some insights into the above as I’m a bit lost on how > this is actually expected to work, I’ve been reading both RabbitMQ and Camel > docs but I’m none the wiser of how to do the DR flip-over or if that’s even > possible to do. I really hope it’s just some trivial config thing we’ve been > missing so far. > > > > > > Thanks, > > Valdis > > Vhi Group DAC (Vhi) is a holding company for insurance and healthcare > services, which include Vhi Healthcare DAC, Vhi Insurance DAC, Vhi > Health Services DAC and Vhi Investments DAC. Vhi Healthcare DAC > trading as Vhi Healthcare and Vhi Insurance DAC trading as Vhi > Insurance are regulated by the Central Bank of Ireland. Vhi Healthcare > is tied to Vhi Insurance DAC for health insurance in Ireland which is > underwritten by Vhi Insurance DAC. Vhi Healthcare is tied to Zurich > Life Assurance plc for Vhi Life Term Insurance and Vhi Mortgage > Protection which is underwritten by Zurich Life Assurance plc. Vhi > Healthcare is tied to Collinson Insurance Services Limited for > MultiTrip Travel Insurance, Backpacker Travel Insurance and Vhi Dental > Insurance which are underwritten by Great Lakes Insurance SE, UK > branch and for Vhi Canada Cover and Vhi International Health Insurance > which are underwritten by Astrenska Insurance Limited. For more > information about the Vhi Group please go to: > https://scanmail.trustwave.com/?c=6600&d=nf2t3GToGb7gdYOLSUghL0uPKJIB3 > Fxo7KRLwqE7Vw&s=33&u=https%3a%2f%2fwww%2evhi%2eie%2fabout-vhi > > > Tá Vhi Group DAC (Vhi) ina chuideachta sealbhaíochta le haghaidh > seirbhísí árachais agus seirbhísí cúram sláinte, lena n-áirítear Vhi > Healthcare DAC, Vhi Insurance DAC, Vhi Health Services DAC agus Vhi > Investments DAC. Déanann Banc Ceannais na hÉireann rialáil ar Vhi > Healthcare DAC, ag trádáil dó mar Vhi Healthcare, agus ar Vhi > Insurance DAC, ag trádáil dó mar Vhi Insurance. Tá Vhi Healthcare > ceangailte le Vhi Insurance DAC le haghaidh árachas sláinte in Éirinn, > rud atá frithgheallta ag Vhi Insurance DAC. Tá Vhi Healthcare > ceangailte le Zurich Life Assurance plc le haghaidh Árachais Saoil de > chuid Vhi agus Árachas Cosanta Morgáiste de chuid Vhi atá > frithgheallta ag Zurich Life Assurance plc. Tá Vhi Healthcare > ceangailte le Collinson Insurance Services Limited le haghaidh Árachas > Taistil Ilturais agus Turasóirí Mála Droma agus Árachas Fiaclóireachta > de chuid Vhi atá frithgheallta ag Great Lakes Insurance SE, UK branch > agus le haghaidh Clúdach Cheanada de chuid Vhi agus Árachas Sláinte > Idirnáisiúnta de chuid Vhi atá frithgheallta ag Astrenska Insurance > Limited. Chun tuilleadh faisnéise a fháil faoi Ghrúpa Vhi, tabhair > cuairt ar: > https://scanmail.trustwave.com/?c=6600&d=nf2t3GToGb7gdYOLSUghL0uPKJIB3 > Fxo7KRLwqE7Vw&s=33&u=https%3a%2f%2fwww%2evhi%2eie%2fabout-vhi > > This e-mail and any files transmitted with it contain information which may > be confidential and which may also be privileged and is intended solely for > the use of the individual or entity to whom it is addressed. Unless you are > the intended recipient you may not copy or use it, or disclose it to anyone > else. Any opinions expressed are that of the individual and not necessarily > that of the Vhi Group. If you have received this e-mail in error please > notify the sender by return. > > > > > > > -- Claus Ibsen ----------------- http://scanmail.trustwave.com/?c=6600&d=nf2t3GToGb7gdYOLSUghL0uPKJIB3Fxo7KdDx69vVg&s=33&u=http%3a%2f%2fdavsclaus%2ecom @davsclaus Camel in Action 2: https://scanmail.trustwave.com/?c=6600&d=nf2t3GToGb7gdYOLSUghL0uPKJIB3Fxo7KhNkqU5BQ&s=33&u=https%3a%2f%2fwww%2emanning%2ecom%2fibsen2
