Hi all,
We have been investigating a graceful shutdown issue with
camel-azure-servicebus (ServiceBusComponent) and would like to share our
findings and get your input on whether this is a known issue or a bug.
Environment:
- Camel version: 4.17.0
- Spring Boot 3.5.11 with camel-spring-boot
- Azure Service Bus
- Kubernetes (AKS), pods receive SIGTERM on rolling deployment
SYMPTOMS
When a pod receives SIGTERM while a message is actively being processed, the
following errors appear in logs after the message is processed successfully:
WARN ServiceBusConsumer: Error during processing exchange. Exchange[...]
Caused by: [java.util.concurrent.RejectedExecutionException - null]
java.util.concurrent.RejectedExecutionException
at
o.a.c.p.e.RedeliveryErrorHandler$SimpleTask.runNotAllowed(RedeliveryErrorHandler.java:469)
at
o.a.c.p.e.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:402)
at
o.a.c.p.e.RedeliveryErrorHandler$SimpleTask.done(RedeliveryErrorHandler.java:392)
at o.a.c.component.bean.MethodInfo$1.doProceed(MethodInfo.java:353)
at o.a.c.component.bean.MethodInfo$1.proceed(MethodInfo.java:286)
at
o.a.c.component.bean.AbstractBeanProcessor.process(AbstractBeanProcessor.java:88)
at
o.a.c.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
at o.a.c.processor.Pipeline.process(Pipeline.java:162)
at
o.a.c.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:353)
at
o.a.c.component.azure.servicebus.ServiceBusConsumer.processMessage(ServiceBusConsumer.java:85)
at
com.azure.messaging.servicebus.MessagePump.notifyMessage(MessagePump.java:163)
ERROR ServiceBusReceiverAsyncClient: Cannot perform operation 'abandoned' on
a disposed receiver.
java.lang.IllegalStateException: Cannot perform operation 'abandoned' on a
disposed receiver.
at
o.a.c.component.azure.servicebus.ServiceBusConsumer$ConsumerOnCompletion.onFailure(ServiceBusConsumer.java:202)
at o.a.c.impl.engine.DefaultUnitOfWork.done(DefaultUnitOfWork.java:269)
at
o.a.c.impl.engine.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:959)
The message was processed successfully — the processor finished successfully,
the downstream topic was published — but the ACK never reached ASB. The message
is left unacknowledged and redelivered, causing duplicate processing.
OUR ASSUMPTION — SUSPECTED ROOT CAUSE
Assumption is that there is some issue with the shutdown orchestration of
DefaultShutdownStrategy, which shuts down underlying ASB clients as soon as
SIGTERM is received while it correctly detects that there are inflight messages
and waits for all of them to finish. However, when they finish, since
underlying ASB layer is already shut down, it can’t send ACK to the ASB
service, causing message to be redelivered once message lock expires.
We were expecting that once SIGTERM signal is received, it should only prevent
consumption of new messages, not block in-flight exchanges from completing
their pipeline.
QUESTIONS FOR THE COMMUNITY
1. Is this a known issue with ServiceBusComponent or DefaultShutdownStrategy?
3. Is there a recommended configuration or workaround for ServiceBusComponent
consumers to ensure ACK is sent after successful processing during graceful
shutdown?
If any additional logs or information is required, please feel free to let me
know!
Thanks,
Alex
------------------------------------------------------------------------
Disclaimer The information in this email and any attachments may contain
proprietary and confidential information that is intended for the addressee(s)
only. If you are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution, retention or use of the contents of this
information is prohibited. When addressed to our clients or vendors, any
information contained in this e-mail or any attachments is subject to the terms
and conditions in any governing contract. If you have received this e-mail in
error, please immediately contact the sender and delete the e-mail.
