Simon Rasmussen created CAMEL-21659:
---------------------------------------
Summary: Improve aws sqs TimeoutExtender by introducing deadline
concept to messages
Key: CAMEL-21659
URL: https://issues.apache.org/jira/browse/CAMEL-21659
Project: Camel
Issue Type: Improvement
Components: camel-aws2-sqs
Affects Versions: 4.8.3
Reporter: Simon Rasmussen
Assignee: Simon Rasmussen
After upgrading to 4.8.3, we I started seeing the following warnings in our
logs, quite often:
{noformat}
2025-01-27T06:54:34.226Z
WARN 11 --- [X] [TimeoutExtender]
o.a.c.component.aws2.sqs.Sqs2Consumer ::
[BatchResultErrorEntry(Id=FBDB3014CD64A96-0000000000093816,
SenderFault=true, Code=ReceiptHandleIsInvalid, Message=Message does
not exist or is not available for visibility timeout change)]{noformat}
The error occurs when an attempt to extend message visibility for a message
that has already been deleted in SQS.
In CAMEL-19656, the implementation was switched to using the batched call, but
failed to implement the proper ignore in case of above error – as the old
implementation had.
I also noticed that the current implementation extends ALL messages in the
queue of non-processed – not just those that are about to expire. This happens
because the TimeoutExtender, currently will make a copy of all the values in
the Map, and extend them, in batches.
In case of 100 messages, this will require 10 requests, even though the last
messages in the queue might not need to be extended yet.
If the route is used together with the threads EIP, then the default queue size
is 1000, which makes above algorithm really inefficient.
I suggest introducing a concept of a deadline to each entry in the
TimeoutExtender. The deadline is set to now() plus 80% of the
visibilityTimeout. When sending requests, only messages that will expire before
the next execution of the TimeoutExtender will be extended. The TimeoutExtender
task itself should run more frequently, but will pick up less messages each
time. I suggest it running every visibilityTimeout/2. In a typical
configuration, this would result in executing TimeoutExtender every 10 seconds.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
