shashank created CAMEL-25558:
--------------------------------
Summary: camel-aws2-s3 - a listed object that cannot be got
(deleted meanwhile, or a failed get) fails the poll and stays in the
in-progress repository, so it is never consumed until a restart
Key: CAMEL-25558
URL: https://issues.apache.org/jira/browse/CAMEL-25558
Project: Camel
Issue Type: Bug
Components: camel-aws2-s3
Reporter: shashank
Assignee: shashank
When the consumer lists the bucket, {{AWS2S3Consumer.createExchanges(List)}}
adds each listed key to the endpoint's {{inProgressRepository}} (line 264 at
main 3a6cdba883a5) and then gets the object (line 287). When that get throws,
the catch block (lines 301-315) closes the streams and removes the keys of the
exchanges that were already created, and rethrows, so the poll fails. The key
of the object that failed has no exchange yet, so it is not removed. The
default repository is a memory repository that is only cleared when the
endpoint stops, so every later poll skips that key as "already in progress"
(lines 264-268).
Effects:
* A listed object that is deleted before the get (another consumer on the same
bucket, a lifecycle rule, or the exchange of an earlier poll that completes
after the listing, with an asynchronous route) makes the get fail with
{{NoSuchKeyException}}: that poll fails. The other listed objects are consumed
by a following poll (with a truncated listing the marker has already moved on,
lines 173-188, so they wait until the listing starts over). A new object stored
later under the same key is never consumed.
* An object whose get fails for another reason (403, throttling, a network
error), or whose download fails while it is read into the body
({{includeBody=true}}, line 500), fails that poll in the same way, stays in the
bucket, and is never consumed again, until the endpoint is restarted. In the
second case the exchange that was created for it (line 493) is also never
released to the exchange factory.
This is not new and not caused by CAMEL-25521: camel-4.22.x has the same loop
and catch block.
For comparison: camel-minio (CAMEL-25512) and camel-google-storage
(CAMEL-25519) skip a listed object that no longer exists (DEBUG) and consume
the rest; camel-ibm-cos skips a listed object whose get fails for any reason
(WARN with the key and the message, no stack trace), removes its key and
continues with the rest.
h3. Reproduction
New {{AWS2S3ConsumerDeletedObjectTest}} (6 tests) with the in-process
{{FakeS3Server}} of the module (it can now remove an object right after a
listing returned it, refuse the gets of an object with 403, and cut the body of
a get short), and {{AWS2S3ConsumerDeletedObjectPooledExchangeTest}}, which runs
the same 6 with {{PooledExchangeFactory}} and adds one. The scheduler is not
started ({{startScheduler=false}}); the tests call {{poll()}}. On main all 13
fail:
* {{listedObjectThatNoLongerExistsIsSkipped}}: {{a.txt}}, {{b.txt}}, {{c.txt}};
{{b.txt}} is removed after the listing. Expected: the poll returns 2 and routes
a and c with {{CamelBatchSize}} 2. On main: {{NoSuchKeyException: The specified
key does not exist}} from {{poll()}}.
* {{objectCreatedAgainAfterItWasSkippedIsConsumed}}: after that poll a new
{{b.txt}} is stored and must be consumed by the next poll. On main:
{{NoSuchKeyException}} from the first poll.
* {{objectWhoseGetIsRefusedDoesNotStopTheOtherObjects}}: every get of {{b.txt}}
is refused. Expected: the poll returns 2 (a and c), the next poll consumes a
new {{d.txt}} and tries {{b.txt}} again, and {{b.txt}} is consumed once its get
is allowed. On main: {{S3Exception: Access Denied}} from the first {{poll()}}.
* {{skippedObjectOfATruncatedListingIsTriedAgainWhenTheListingStartsOver}}
({{maxMessagesPerPoll=2}}, four objects, {{b.txt}} refused): the polls return
1, 2 (second page, {{b.txt}} not tried), 0 (listing starts over, {{b.txt}}
tried again), then 1 once the get is allowed. On main: {{S3Exception}} from the
first {{poll()}}.
* {{objectWhoseDownloadFailsIsSkippedAndNotLeftInProgress}}: the content stream
throws an {{IOException}} while it is read into the body. Expected: the poll
returns 0, the stream is closed, the key is not in progress. On main:
{{RuntimeCamelException}} ({{IOException: connection reset}}) from {{poll()}}.
* {{objectStreamIsClosedWhenIncludeS3ObjectFails}}: an overridden
{{includeS3Object}} throws. Expected as above. On main:
{{IllegalStateException}} from {{poll()}}.
* {{exchangeOfAnObjectWhoseDownloadFailsIsGivenBackToThePool}} (pooled): the
body of {{b.txt}} is cut short. Expected: a, c and d are routed with their own
bodies, 4 exchanges taken from the pool and 4 released. On main:
{{RuntimeCamelException}} ({{ConnectionClosedException: Premature end of
Content-Length delimited message body}}) from {{poll()}}.
h3. Proposed fix
In {{createExchanges(List)}}: a failure to get one listed object, or to create
its exchange, no longer fails the poll. The object is skipped (stream closed,
exchange given back to the exchange factory, key removed from the in-progress
repository), the other listed objects are consumed in the same poll, the
skipped object is not counted, and a later poll tries it again.
{{NoSuchKeyException}} is logged at DEBUG; any other failure at WARN with the
key and the message, without a stack trace, because an object that can never be
read (archived, access denied) is logged at every poll that lists it. This is
what camel-ibm-cos does; the alternative (keep failing the poll and only remove
the key) would make such an object fail every poll and block the objects listed
after it.
Consequence: such a failure is no longer a failed poll, so it does not reach
the exception handler ({{bridgeErrorHandler}}) and does not count for
{{backoffErrorThreshold}}. The listing marker is set from the listing response
before the gets, as before: with a truncated listing a skipped object is tried
again when the listing starts over, otherwise at the next poll.
Not changed, known limits: with {{fileName}}, a failed get still fails the
poll. An {{Error}} (for example {{OutOfMemoryError}} while a large object is
read into the body) is not caught, so its key still stays in progress.
With the pooled exchange factory the exchange of a skipped object is given back
with {{done()}}: {{releaseExchange(exchange, true)}} on an exchange that is not
done adds it to the pool twice, and two later objects of the poll then share it
(seen with the pooled test). The same call for never-routed exchanges exists on
main in this consumer ({{processBatch}} when stopping, the outer catch) and in
other consumers; not changed here.
No new option. 4.23 upgrade-guide note under the existing camel-aws2-s3
in-progress entry. camel-aws2-s3 unit tests with the fix: 98, 0 failures (the
68 integration tests need LocalStack and were not run).
Found by comparing the consumer with the other object storage consumers after
CAMEL-25521, then confirmed with the real consumer as above.
Affected: main; camel-4.22.x, camel-4.18.x and camel-4.14.x have the same loop
and catch block (read, not run).
Duplicate check (2026-10-11): JIRA text "aws2-s3 NoSuchKey" (1: CAMEL-25521,
the related fix, which does not change this path), "AWS2S3Consumer
NoSuchKeyException" (1: CAMEL-17110, which added the in-progress repository),
component camel-aws2-s3 with text "inProgressRepository" (1: CAMEL-25521),
component camel-aws2-s3 created since 2026-01-01 (10 issues, none about a
deleted object or a failed get in the consumer); GitHub pull requests
"AWS2S3Consumer NoSuchKey" and "aws2-s3 NoSuchKey": none; no open pull request
touches {{AWS2S3Consumer}}.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
