shashank created CAMEL-25299:
--------------------------------
Summary: Error handler - with asyncDelayedRedelivery,
allowRedeliveryWhileStopping=false is ignored: a route stop waits for the
redelivery delay and then redelivers
Key: CAMEL-25299
URL: https://issues.apache.org/jira/browse/CAMEL-25299
Project: Camel
Issue Type: Bug
Components: camel-core
Reporter: shashank
The documentation of the dead letter channel says that with
{{allowRedeliveryWhileStopping=false}} any new redelivery attempt is not
allowed once the stop of the route (or Camel) has been triggered: a
{{RejectedExecutionException}} is set and the exchange stops being routed (the
dead letter channel still receives it).
That holds for the synchronous redelivery: {{RedeliveryTask.sleep()}} waits in
chunks of 1 second and checks {{preparingShutdown}} after each chunk, then
marks the exchange rejected and exhausted, so it goes to the dead letter
channel within a second
({{NotAllowRedeliveryWhileStoppingDeadLetterChannelTest}}).
With {{asyncDelayedRedelivery}} the delay is a task on the error handler's
scheduled thread pool:
{code:java}
executorService.schedule(() -> reactiveExecutor.schedule(this::redeliver),
redeliveryDelay, TimeUnit.MILLISECONDS);
{code}
and {{redeliver()}} calls the output again without any check of the shutdown
state (only {{run()}} checks {{isRunAllowed()}}, and the redelivery decision in
{{doRun()}} was taken before the stop). So when the route is stopped while a
redelivery waits for its delay:
* the graceful stop waits for the inflight exchange for the whole remaining
delay, or until the shutdown timeout (45 s by default) if the delay is longer,
then stops the consumer forcibly;
* when the delay ends, the message is redelivered anyway (also after the forced
stop, into the stopped route), which is what
{{allowRedeliveryWhileStopping=false}} is meant to prevent.
CAMEL-3364 (2.11.0), which added {{allowRedeliveryWhileStopping}}, already
named this case in its discussion: the error handler keeps the delayed
exchanges as tasks in its thread pool, and on shutdown those have to be dealt
with too. The implementation only made {{sleep()}} check the shutdown state;
nothing in the ticket or the code says the scheduled task was meant to be
exempt.
h3. Reproduction
Test {{NotAllowRedeliveryWhileStoppingAsyncDelayedTest}} (camel-core):
{{deadLetterChannel("mock:dead").maximumRedeliveries(5).redeliveryDelay(1 hour,
capped to the 60 s
maximumRedeliveryDelay).asyncDelayedRedelivery().allowRedeliveryWhileStopping(false)}},
a route {{seda:start -> mock:foo -> throwException}}, and a scheduled thread
pool for the error handler that counts down a latch when the redelivery is
scheduled. After the first attempt failed and the redelivery is scheduled, the
test calls {{stopRoute("foo")}} (shutdown timeout 5 s). Expected: the dead
letter channel gets the exchange with {{RejectedExecutionException("Redelivery
not allowed while stopping")}}, no redelivery, nothing inflight. A second test
does the same with {{context.stop()}}. On main both stops run into the 5 s
timeout and the dead letter channel has nothing:
{noformat}
NotAllowRedeliveryWhileStoppingAsyncDelayedTest.testStopRouteRejectsScheduledRedelivery
mock://dead Received message count. Expected: <1> but was: <0>
NotAllowRedeliveryWhileStoppingAsyncDelayedTest.testStopCamelContextRejectsScheduledRedelivery
mock://dead Received message count. Expected: <1> but was: <0>
{noformat}
The defect was found with a TLA+ model of the redelivery task (delivery,
{{run()}}/{{doRun()}}, synchronous {{sleep()}}, the scheduled task,
{{redeliver()}}) and the graceful shutdown ({{prepareShutdown}}, waiting for
inflight exchanges, timeout). The invariant "no redelivery starts after its
delay ended while the error handler prepares to shut down
(allowRedeliveryWhileStopping=false)" is violated with
{{asyncDelayedRedelivery}} in 6 steps (deliver, fail, schedule,
prepareShutdown, delay ends, redeliver); it holds for the synchronous
redelivery, with and without a dead letter channel, and for the fixed task.
h3. Proposed fix
When the redelivery is not allowed while stopping, the scheduled task wakes up
every second, like {{sleep()}}, and when the error handler is preparing to shut
down it rejects the redelivery the same way
({{RejectedExecutionException("Redelivery not allowed while stopping")}},
redelivery exhausted, back to {{run()}}, which moves the exchange to the dead
letter channel or fails it). When the delay is over it redelivers as before.
With {{allowRedeliveryWhileStopping=true}} (the default) nothing changes: one
task for the whole delay. If the thread pool rejects the next wake-up, the
redelivery is rejected the same way instead of losing the callback.
With the fix the new tests pass (each in about 1.5 s), and {{*Redelivery*}},
{{NotAllowRedelivery*}} and {{DeadLetterChannel*}} tests in camel-core pass (95
tests).
Cost: one short task per second per waiting redelivery, only with
{{allowRedeliveryWhileStopping=false}} (the same rate as the synchronous
{{sleep()}} loop, but without a blocked thread). One difference to {{sleep()}}:
{{sleep()}} does not check at all for a delay below 1 second, while the
scheduled task checks once when such a delay ends, so a short redelivery whose
delay ends while stopping is rejected too (which is what the option describes).
The upgrade guide gets a note, as a stop now rejects these messages within a
second instead of redelivering them.
Not addressed (low): with {{allowRedeliveryWhileStopping=true}} (the default),
when Camel itself is stopped and the shutdown times out, the error handler
thread pool is shut down with the delayed task still in it, so the exchange is
never completed; Camel is stopping and the await manager releases waiting
callers. (With {{false}} the fix rejects the task within a second, before the
timeout.)
Affected: 4.14.x, 4.18.x and main (same code).
Duplicate check (2026-10-03): JIRA text "allowRedeliveryWhileStopping"
(CAMEL-6033, CAMEL-20744, CAMEL-3364, CAMEL-19896, CAMEL-12603),
"asyncDelayedRedelivery" with stop/shutdown (none), "redelivery" with
"stopping" since 2022 (11 issues, none about the asynchronous delay). GitHub
pull requests "asyncDelayedRedelivery" (#1771, a doc note, 2017),
"allowRedeliveryWhileStopping" (#23179, CAMEL-23494).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
