shashank created CAMEL-25700:
--------------------------------
Summary: camel-aws2-sqs - with extendMessageVisibility=true and
maxMessagesPerPoll > 1, the messages that wait in the batch are not extended
and are delivered a second time
Key: CAMEL-25700
URL: https://issues.apache.org/jira/browse/CAMEL-25700
Project: Camel
Issue Type: Bug
Components: camel-aws2-sqs
Reporter: shashank
Assignee: shashank
With {{extendMessageVisibility=true}} the SQS consumer keeps the messages it is
processing invisible: a background task ({{TimeoutExtender}}) calls
{{ChangeMessageVisibilityBatch}} for every message it knows about before its
visibility timeout runs out.
A poll receives up to {{maxMessagesPerPoll}} messages at once, and their
visibility timeout starts at that receive. {{Sqs2Consumer.processBatch}} then
routes them one after the other, and registers each message with the extender
only when its own turn comes ({{timeoutExtender.add(exchange)}} inside the
loop, line 151 on main d46d51c24550). While the first messages are processed
(synchronous route), the ones that wait behind them are not extended at all.
When the processing of the earlier messages takes longer than
{{visibilityTimeout}}, SQS makes the waiting messages visible again and
delivers them to another consumer (or to the next poll of this one): they are
processed twice, and the delete of the first copy uses a receipt handle that is
no longer current.
This is the scenario of CAMEL-10587 (2016, camel-aws):
{{maxMessagesPerPoll=10}}, {{extendMessageVisibility=true}},
{{visibilityTimeout=30}}, a slow route, "after 30 seconds only 1 message is in
flight". That ticket was closed as Abandoned; its only commit (c6a15565acdf,
2017) started the extender's executor before the first poll; the per-message
registration stayed in the loop, and camel-aws2-sqs (and the batched extender
of CAMEL-19656) kept it.
Trigger: {{extendMessageVisibility=true}} (the option users set because their
processing is slow) with {{maxMessagesPerPoll}} greater than 1 and a
synchronous route where the messages before the last one together take longer
than {{visibilityTimeout}}. Example: 10 messages per poll, 10 seconds each,
{{visibilityTimeout=30}}: at least messages 4 to 10 are delivered again (2 and
3 are extended only at about the moment they expire). With the default
{{maxMessagesPerPoll}} (one message per poll) nothing changes.
h3. Reproduction
New {{SqsConsumerExtendMessageVisibilityBatchTest}} with the module's
{{AmazonSQSClientMock}} (no AWS): two messages on the queue,
{{maxMessagesPerPoll=2&visibilityTimeout=2&extendMessageVisibility=true}}, the
route holds the first message until the extender has sent a
{{ChangeMessageVisibilityBatch}} entry for the receipt handle of the second (at
most 10 seconds). On main it never does:
{noformat}
[the visibility of the second message of the batch is extended while the first
message is processed (otherwise SQS makes it visible again and delivers it a
second time)]
Expecting value to be true but was false
{noformat}
h3. Proposed fix
{{processBatch}} registers every message of the batch with the extender before
it routes the first one, and removes the messages it does not route (the
consumer is stopping or suspending, or the processor threw) so that they become
visible again as they do today. The extender's own rule (extend an entry when
its deadline is near) is unchanged, so a message that is processed quickly is
still not extended.
A second new test suspends the route while the first message is processed: the
second message is not routed and must not be extended afterwards (it passes on
main, where the message is never registered; it guards the removal).
camel-aws2-sqs unit tests with the fix: 77 run, 0 failures, 0 skipped. Not run:
the integration tests (LocalStack), a real queue.
How it was found: reading the ack and visibility paths of the SQS consumer in a
sweep of acknowledge/commit/delete timing; the test confirmed it.
Affected: main (tested). camel-4.23.x, camel-4.22.x and camel-4.18.x register
the message with the extender inside the processing loop the same way (line
151; read, not run).
Duplicate check (2026-10-16): JIRA text "extendMessageVisibility" (CAMEL-10587,
CAMEL-18780, CAMEL-17156, CAMEL-5944, CAMEL-16956: none open about the batch),
"TimeoutExtender" / "visibility extender" (CAMEL-19562, CAMEL-19656,
CAMEL-21550, CAMEL-21659, CAMEL-21769: start delay, batching, deadlines, an
error that extended forever; none about the waiting messages of a batch);
GitHub pull requests "sqs extend", "Sqs2Consumer visibility" (#9108, #10828,
#17262, #16592, #16945, #11102; all merged, none changes where a message is
registered).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
