Croway opened a new pull request, #29274: URL: https://github.com/apache/camel/pull/29274
JIRA: [CAMEL-26662](https://issues.apache.org/jira/browse/CAMEL-26662), [CAMEL-26661](https://issues.apache.org/jira/browse/CAMEL-26661) ## What was wrong - `AWS2S3Consumer.createExchanges` sent a GetObject for every listed key of the poll, and with `includeBody=true` read each body into a `byte[]`, before `processBatch` routed the first exchange. The whole batch was in memory, and the first exchange waited for the last download. - With `includeBody=false`, the streams of all the objects of the poll stayed open, and each held a pooled connection. The client Camel builds has the AWS SDK default pool of 50 connections. With `maxMessagesPerPoll` above 50, the 51st GetObject waited 4 x 10 s for a connection. By then the earlier streams were older than 60 s, so the SDK had aborted them (`ResponseInputStream`, since SDK 2.32.32), and most exchanges failed with `Attempted read on closed stream`. The failed objects were not deleted, so every poll repeated this (CAMEL-26661). - With `ignoreBody=true`, the consumer and the producer `getObject` sent a GetObject and closed the stream without reading it. With the Apache HTTP client, `close()` reads the rest of the body so that the connection can be reused, so the whole object was downloaded anyway. ## What changed - `poll()` only lists the objects (and adds them to the in-progress repository). `processBatch` gets each listed object right before its exchange is routed. So only the objects of the exchanges in flight are in memory or hold a connection. - The CAMEL-25558 handling is kept as it was: - an object deleted after the listing is skipped; - an object that cannot be got is skipped and given to the exception handler, now once the other exchanges of the poll were routed; - the poll fails only when none of the listed objects could be got; - `fileName` still fails the poll on any failure. - **Batch properties:** `CamelBatchSize` / `CamelBatchIndex` / `CamelBatchComplete` now count the listed objects. A skipped object leaves a gap, as camel-minio and camel-file already do. The value returned by the poll still counts the routed exchanges, which keeps `greedy` and `sendEmptyMessageWhenIdle` unchanged. The other options were a HEAD request per object before the GET, which doubles the requests, or looking one object ahead, which keeps two streams open and brings back the 60 s abort. - **ignoreBody=true:** the consumer (listing and `fileName`) and the producer `getObject` (plain and POJO request) now send HeadObject. HeadObject returns the same metadata fields, and the same headers are set. The new `AWS2S3Utils.toHeadObjectRequest` copies every field of the `GetObjectRequest`; the two requests have the same members in SDK 2.57.7. A 404 on HEAD is still a `NoSuchKeyException` (`ExceptionTranslationInterceptor`), so the skip logic is unchanged. - **Deprecated `maxConnections`:** it is still deprecated with no effect. With the lazy get, the consumer no longer needs a larger pool. Its description used to point to a non-existent `httpClientBuilder` option. It now says that the pool is the SDK default of 50, and that a different size needs your own `amazonS3Client`. - The protected `AWS2S3Consumer.createExchanges(List<S3Object>)` is removed. This is noted in the upgrade guide. ## Tests New tests use the existing `FakeS3Server` (JDK `HttpServer`) with the S3 client Camel builds, so they go through the real Apache5 pool: - `AWS2S3ConsumerGetEachObjectTest` covers: - the GET-then-route order; - the CAMEL-26661 regression with 60 objects, `includeBody=false` and `maxMessagesPerPoll=60`; - the batch properties with a skipped object; - `ignoreBody` with HEAD for the listing and for `fileName`; - the skip, exception-handler and fail-the-poll behaviour with HEAD. - `AWS2S3ProducerGetObjectIgnoreBodyTest` covers `getObject` with `ignoreBody`, plain and POJO. - `AWS2S3UtilsTest` covers the request conversion. - Existing tests: - `AWS2S3ConsumerDeletedObjectTest` now expects `CamelBatchSize` 3, and starts the consumer in the two tests that poll a hand-built consumer. - `AWS2S3ConsumerInProgressTest` also checks that a stopping consumer does not get the objects it does not route. Evidence: - **On main (39039a5ce040):** 7 of the 11 new consumer/producer tests fail. - Gets seen when routing were `[3, 3, 3]` instead of `[1, 2, 3]`. - The 60-object poll took 60 s and returned 59, and 48 routed exchanges failed with `Attempted read on closed stream`. - With `ignoreBody`, 1 to 2 GETs were sent instead of 0. - **With the fix:** all pass. - Module unit tests: 143 run, 0 failures. - Module ITs on Floci (`mvn verify`): 68 run, 0 failures, 9 manual ITs skipped. - **The earlier reproducer, adapted:** - The 52-object `includeBody=false` poll now takes 0.26 s with no failure. - A 64 MiB object with `ignoreBody` now transfers 0 bytes (HEAD), for the consumer and the producer. - A 61 s route on the first object no longer aborts the second. ## Docs and upgrade guide - `aws2-s3-component.adoc` (Batch Consumer): describes the per-object get, HEAD for `ignoreBody`, and the meaning of the batch properties. - `aws2-s3-consumer-examples.adoc`: the `ignoreBody` sentence now matches the behaviour, and notes the SDK 60 s abort of an unread stream. - Option descriptions of `ignoreBody` and `maxConnections`; the generated JSON, catalog copies and endpoint/component DSL are regenerated. - `camel-4x-upgrade-guide-4_24.adoc`: new entry about: - the per-object get and the batch properties; - the later exception-handler call; - the removed protected method; - HEAD for `ignoreBody`: archived (Glacier) objects are now consumed, and a failed HEAD has no S3 error code except `NoSuchKey`. ## Backport - **camel-4.22.x:** candidate. It has the same consumer code, including CAMEL-25558, so this should port mechanically, but it needs the upgrade-guide entry for the 4.22.x line because the batch properties and `ignoreBody` requests change. - **camel-4.18.x:** it has the CAMEL-26661 bug, but not CAMEL-25558, so it would not be a straight cherry-pick. The workaround there is a user-supplied `S3Client` with a larger pool. _Claude Code on behalf of Federico Mariani_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
