Hello everyone. I hope to get some useful information about our current problem. I'll try to keep it brief and still provide enough information. Ask if any more information is required.
We migrated our camel routes which receive messages from RabbitMQ to "camel spring rabbitmq" (since "camel rabbitmq" has been removed in camel 4). First of all: <spring-boot.version>3.2.4</spring-boot.version> <camel.version>4.4.0</camel.version> <java.version>17</java.version> (First we also tried with camel 3.22.1 and spring boot 2.7.18) Everything worked great after the migration, except for one special case: For maintenance we sometimes need to STOP the routes (e.g. via JMX): The rabbitmq consumers should be stopped and inflight messages remain in RabbitMQ, ready to be consumed again after maintenance. That worked like a charm with "camel rabbitmq component": Even when there was a message in (Camel) RETRY for some minutes, that route has been stopped after the configured timeout of 5 seconds, the retry (Thread.sleep) has been interrupted and the message in RabbitMQ was ready to be consumed again. But with "camel spring rabbitmq" the message in retry is no longer interrupted: It continues to run despite the rabbit consumer being and the consuming route has been stopped (we can see that in Hawt.io) I attached two log outputs: The first one with "camel rabbitmq" (where you can see the interruption of Thread.sleep) and another one with "camel spring rabbitmq": As you can see, camel is waiting for the inflight message to finish but then forces the route to stop ... but the retry thread is still running and tries to redeliver the message (we can see the delivery failures while the consumer and route is stopped) Thanks for reading all this. I hope I made my point clear?! With kind regards Bert
20:43:39.314 DEBUG Starting to graceful shutdown 1 routes (timeout 3 seconds) - o.a.c.i.e.DefaultShutdownStrategy [Thread:qtp476221290-456] 20:43:39.315 DEBUG There are 1 routes to shutdown - o.a.c.i.e.DefaultShutdownStrategy [Thread:Camel (Mesh) thread #90 - ShutdownTask] 20:43:42.315 WARN Timeout occurred during graceful shutdown. Forcing the routes to be shutdown now. Notice: some resources may still be running as graceful shutdown did not complete successfully. - o.a.c.i.e.DefaultShutdownStrategy [Thread:qtp476221290-456] 20:43:42.315 ERROR Thread Interrupted! - o.a.c.c.rabbitmq.RabbitConsumer [Thread:Camel (Mesh) thread #90 - ShutdownTask] 20:43:42.315 DEBUG Closing connection: amqp://mesh-oge@127.0.0.1:5672/mesh-oge with timeout: 30000 ms. - o.a.c.c.rabbitmq.RabbitMQConsumer [Thread:Camel (Mesh) thread #90 - ShutdownTask] 20:43:42.319 DEBUG Forcing shutdown of ExecutorService: org.apache.camel.util.concurrent.RejectableThreadPoolExecutor@1aca7851[Running, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 1][rabbitmq://TSO06-to-TSO-Retry?exchangeType=direct&queue=TSO06-to-TSO-Retry] - o.a.c.i.e.BaseExecutorServiceManager [Thread:qtp476221290-456] 20:43:42.319 DEBUG Route: TSO06-to-TSO-Retry suspended and shutdown deferred, was consuming from: rabbitmq://TSO06-to-TSO-Retry - o.a.c.i.e.DefaultShutdownStrategy [Thread:Camel (Mesh) thread #90 - ShutdownTask] 20:43:42.319 DEBUG Unregistered MBean with ObjectName: org.apache.camel:context=Mesh,type=threadpools,name="RabbitMQEndpoint(0x327ab024)" - o.a.c.m.DefaultManagementAgent [Thread:qtp476221290-456] 20:43:42.319 DEBUG Graceful shutdown of 1 routes completed in 3s4ms - o.a.c.i.e.DefaultShutdownStrategy [Thread:qtp476221290-456] 20:43:42.319 DEBUG Route: TSO06-to-TSO-Retry shutdown complete, was consuming from: rabbitmq://TSO06-to-TSO-Retry - o.a.c.i.e.DefaultShutdownStrategy [Thread:Camel (Mesh) thread #90 - ShutdownTask] 20:43:42.319 DEBUG Unregistered MBean with ObjectName: org.apache.camel:context=Mesh,type=consumers,name=RabbitMQConsumer(0x7bc340e) - o.a.c.m.DefaultManagementAgent [Thread:qtp476221290-456] 20:43:42.320 DEBUG Unregistered MBean with ObjectName: org.apache.camel:context=Mesh,type=producers,name=DirectProducer(0x7075359c) - o.a.c.m.DefaultManagementAgent [Thread:qtp476221290-456] 20:43:42.320 DEBUG Stopping producer: Producer[direct://TSO06-to-TSO-SendRoute] - o.a.camel.support.DefaultProducer [Thread:qtp476221290-456] 20:43:42.320 DEBUG Processing onFailure: Exchange[] - o.a.c.p.OnCompletionProcessor [Thread:Camel (Mesh) thread #87 - rabbitmq://TSO06-to-TSO-Retry] 20:43:42.320 DEBUG Unregistered MBean with ObjectName: org.apache.camel:context=Mesh,type=processors,name="Init [mesh-vms-tso06:WF-TSO06-to-TSO:006]" - o.a.c.m.DefaultManagementAgent [Thread:qtp476221290-456] 20:43:42.320 DEBUG Unregistered MBean with ObjectName: org.apache.camel:context=Mesh,type=processors,name="convertBodyTo13" - o.a.c.m.DefaultManagementAgent [Thread:qtp476221290-456] 20:43:42.320 DEBUG Unregistered MBean with ObjectName: org.apache.camel:context=Mesh,type=processors,name="Prepare [mesh-vms-tso06:WF-TSO06-to-TSO:007]" - o.a.c.m.DefaultManagementAgent [Thread:qtp476221290-456] 20:43:42.320 DEBUG Unregistered MBean with ObjectName: org.apache.camel:context=Mesh,type=processors,name="to174" - o.a.c.m.DefaultManagementAgent [Thread:qtp476221290-456] 20:43:42.320 INFO WorkflowRoutePolicy for mesh-vms-tso06:TSO06-to-TSO (route TSO06-to-TSO-Retry): onStop - Mesh [Thread:qtp476221290-456] 20:43:42.320 INFO Stopped TSO06-to-TSO-Retry (rabbitmq://TSO06-to-TSO-Retry) - o.a.c.i.engine.AbstractCamelContext [Thread:qtp476221290-456] 20:43:42.340 WARN Error processing exchange. Exchange[9F42A22EBBF1881-0000000000000045]. Caused by: [java.lang.InterruptedException - sleep interrupted] - o.a.c.c.rabbitmq.RabbitMQConsumer [Thread:Camel (Mesh) thread #87 - rabbitmq://TSO06-to-TSO-Retry] java.lang.InterruptedException: sleep interrupted at java.base/java.lang.Thread.sleep(Native Method) at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.sleep(RedeliveryErrorHandler.java:1586) at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.doRun(RedeliveryErrorHandler.java:801) at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.run(RedeliveryErrorHandler.java:737) at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:181) at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59) at org.apache.camel.processor.Pipeline.process(Pipeline.java:165) at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:390) at org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:83) at org.apache.camel.support.AsyncProcessorSupport.process(AsyncProcessorSupport.java:41) at org.apache.camel.component.rabbitmq.RabbitConsumer.doHandleDelivery(RabbitConsumer.java:114) at org.apache.camel.component.rabbitmq.RabbitConsumer.handleDelivery(RabbitConsumer.java:89) at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149) at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:104) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) 20:43:42.341 ERROR Consumer org.apache.camel.component.rabbitmq.RabbitConsumer@528a2624 (amq.ctag-seUbsoTnXzDQ9_aN0ur2RQ) method handleDelivery for channel AMQChannel(amqp://mesh-oge@127.0.0.1:5672/mesh-oge,1) threw an exception for channel AMQChannel(amqp://mesh-oge@127.0.0.1:5672/mesh-oge,1) - c.r.c.impl.ForgivingExceptionHandler [Thread:Camel (Mesh) thread #87 - rabbitmq://TSO06-to-TSO-Retry] com.rabbitmq.client.AlreadyClosedException: connection is already closed due to clean connection shutdown; protocol method: #method<connection.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0) at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:261) at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:430) at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:424) at com.rabbitmq.client.impl.recovery.RecoveryAwareChannelN.basicReject(RecoveryAwareChannelN.java:114) at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicReject(AutorecoveringChannel.java:449) at org.apache.camel.component.rabbitmq.RabbitConsumer.doHandleDelivery(RabbitConsumer.java:195) at org.apache.camel.component.rabbitmq.RabbitConsumer.handleDelivery(RabbitConsumer.java:89) at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149) at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:104) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840)
18:55:56.933 DEBUG Starting to graceful shutdown 1 routes (timeout 3 seconds) - o.a.c.i.e.DefaultShutdownStrategy [Thread:qtp1577067350-36] 18:55:56.933 DEBUG There are 1 routes to shutdown - o.a.c.i.e.DefaultShutdownStrategy [Thread:Camel (Mesh) thread #2 - ShutdownTask] 18:55:56.933 DEBUG Shutting down Rabbit listener container - o.a.c.c.s.CamelDirectMessageListenerContainer [Thread:Camel (Mesh) thread #2 - ShutdownTask] 18:55:56.933 DEBUG Shutting down - o.a.c.c.s.CamelDirectMessageListenerContainer [Thread:Camel (Mesh) thread #2 - ShutdownTask] 18:55:56.933 DEBUG Canceling SimpleConsumer [queue=TSO06-to-TSO-Retry, index=0, consumerTag=amq.ctag-cFslg-p-2nKgT04b3GuyTA identity=4e16e6ea] - o.a.c.c.s.CamelDirectMessageListenerContainer [Thread:Camel (Mesh) thread #2 - ShutdownTask] 18:55:56.933 DEBUG All consumers canceled - o.a.c.c.s.CamelDirectMessageListenerContainer [Thread:Camel (Mesh) thread #2 - ShutdownTask] 18:55:59.933 WARN Timeout occurred during graceful shutdown. Forcing the routes to be shutdown now. Notice: some resources may still be running as graceful shutdown did not complete successfully. - o.a.c.i.e.DefaultShutdownStrategy [Thread:qtp1577067350-36] 18:55:59.933 WARN Interrupted waiting for consumers. Continuing with shutdown. - o.a.c.c.s.CamelDirectMessageListenerContainer [Thread:Camel (Mesh) thread #2 - ShutdownTask] 18:55:59.934 DEBUG Shutdown ignored - container is not active already - o.a.c.c.s.CamelDirectMessageListenerContainer [Thread:qtp1577067350-36] 18:55:59.934 DEBUG Shutdown ignored - container is not active already - o.a.c.c.s.CamelDirectMessageListenerContainer [Thread:qtp1577067350-36] 18:55:59.934 DEBUG Route: TSO06-to-TSO-Retry suspended and shutdown deferred, was consuming from: spring-rabbitmq://TSO06-to-TSO-Retry - o.a.c.i.e.DefaultShutdownStrategy [Thread:Camel (Mesh) thread #2 - ShutdownTask] 18:55:59.934 DEBUG Stopping consumer: Consumer[spring-rabbitmq://TSO06-to-TSO-Retry?arg.queue.durable=true&exchangeType=direct&queues=TSO06-to-TSO-Retry] - o.a.camel.support.DefaultConsumer [Thread:qtp1577067350-36] 18:55:59.934 DEBUG Unregistered MBean with ObjectName: org.apache.camel:context=Mesh,type=producers,name=DirectProducer(0x346dcd2f) - o.a.c.m.DefaultManagementAgent [Thread:qtp1577067350-36] 18:55:59.934 DEBUG Stopping producer: Producer[direct://TSO06-to-TSO-SendRoute] - o.a.camel.support.DefaultProducer [Thread:qtp1577067350-36] 18:55:59.934 DEBUG Graceful shutdown of 1 routes completed in 3s1ms - o.a.c.i.e.DefaultShutdownStrategy [Thread:qtp1577067350-36] 18:55:59.934 DEBUG Route: TSO06-to-TSO-Retry shutdown complete, was consuming from: spring-rabbitmq://TSO06-to-TSO-Retry - o.a.c.i.e.DefaultShutdownStrategy [Thread:Camel (Mesh) thread #2 - ShutdownTask] 18:55:59.935 DEBUG Unregistered MBean with ObjectName: org.apache.camel:context=Mesh,type=consumers,name=SpringRabbitMQConsumer(0x3fc19c60) - o.a.c.m.DefaultManagementAgent [Thread:qtp1577067350-36] 18:55:59.935 DEBUG Unregistered MBean with ObjectName: org.apache.camel:context=Mesh,type=processors,name="Init [mesh-vms-tso06:WF-TSO06-to-TSO:006]" - o.a.c.m.DefaultManagementAgent [Thread:qtp1577067350-36] 18:55:59.935 DEBUG Unregistered MBean with ObjectName: org.apache.camel:context=Mesh,type=processors,name="convertBodyTo58" - o.a.c.m.DefaultManagementAgent [Thread:qtp1577067350-36] 18:55:59.935 DEBUG Unregistered MBean with ObjectName: org.apache.camel:context=Mesh,type=processors,name="Prepare [mesh-vms-tso06:WF-TSO06-to-TSO:007]" - o.a.c.m.DefaultManagementAgent [Thread:qtp1577067350-36] 18:55:59.935 DEBUG Unregistered MBean with ObjectName: org.apache.camel:context=Mesh,type=processors,name="to1266" - o.a.c.m.DefaultManagementAgent [Thread:qtp1577067350-36] 18:55:59.935 INFO WorkflowRoutePolicy for mesh-vms-tso06:TSO06-to-TSO (route TSO06-to-TSO-Retry): onStop - Mesh [Thread:qtp1577067350-36] 18:55:59.936 INFO Stopped TSO06-to-TSO-Retry (spring-rabbitmq://TSO06-to-TSO-Retry) - o.a.c.i.engine.AbstractCamelContext [Thread:qtp1577067350-36]
