[
https://issues.apache.org/jira/browse/CAMEL-26661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Federico Mariani updated CAMEL-26661:
-------------------------------------
Description:
h3. Problem
With {{includeBody=false}}, the consumer puts the open {{ResponseInputStream}}
of each object in the body. {{createExchanges}} gets every listed object of the
poll before {{processBatch}} routes the first one
({{components/camel-aws/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Consumer.java}}
lines 201, 298 and 207 on main, commit 7978db3d42dd). So all the streams of a
poll are open at once. Each holds a pooled HTTP connection until its exchange
reads it to the end or closes it.
* The SDK HTTP client pool has 50 connections by default, with a 10 s acquire
timeout ({{SdkHttpConfigurationOption}}, SDK 2.57.7).
* With {{maxMessagesPerPoll}} above 50, the 51st GetObject waits for a
connection: 10 s per attempt, 4 attempts.
* Since SDK 2.32.32, a {{ResponseInputStream}} that is not read within 60 s is
aborted. This is the default of {{ResponseTransformer.toInputStream()}}, which
the consumer uses.
* By the time the waiting GetObject calls give up, the first streams of the
poll are older than 60 s, and their exchanges fail with {{IOException:
Attempted read on closed stream}}.
* The failed objects are neither deleted nor moved, so the next poll lists them
again and the same thing happens on every poll.
* The {{maxConnections}} option (default 60, that is 50 + maxMessagesPerPoll)
has no effect: it is only a getter and a setter, deprecated on main
({{AWS2S3Endpoint.java}} lines 157-162).
* CAMEL-11697 fixed this exact problem in the SDK v1 S3 component by sizing the
pool at 50 + maxMessagesPerPoll. That fix did not carry over to camel-aws2-s3.
h3. How to reproduce
The setup is an in-process fake S3: a JDK {{HttpServer}} serving path-style
ListObjectsV2 and GetObject.
* Consumer:
{{aws2-s3://bkt?overrideEndpoint=true&forcePathStyle=true&uriEndpointOverride=...&includeBody=false&maxMessagesPerPoll=52}}.
* The route converts the body to String.
* The bucket holds 52 objects of 1 byte.
Measured on main with the client Camel builds:
* The poll took 60.2 s.
* The GetObject of the 51st object failed with {{SdkClientException: Unable to
execute HTTP request: Timeout deadline: 10000 MILLISECONDS ... (SDK Attempt
Count: 4)}}. It was skipped and given to the exception handler.
* 49 to 50 of the 51 routed exchanges failed with {{StreamCacheException ...
IOException: Attempted read on closed stream}} (several runs).
Control: the same setup with a user-supplied {{S3Client}} whose pool has 100
connections routed all 52 objects in 0.4 to 0.7 s, with no failure.
Separately, and with any pool size: with 2 objects and a route that takes 61 s
on the first, the second exchange fails the same way.
h3. Proposed change
* Best fix: get each object in {{processBatch}} just before its exchange is
routed, so that only the streams of exchanges in flight are open. This is
CAMEL-26662. It also removes the 60 s abort of the later streams of a poll.
* Otherwise:
** size the pool of the client Camel builds as CAMEL-11697 did (50 +
maxMessagesPerPoll);
** make {{maxConnections}} work again, or remove it;
** document that, with includeBody=false, the routing of one poll must finish
within 60 s.
* Workaround today:
** Supply your own client with a larger pool, through
{{amazonS3Client=#myClient}} or autowiring. Measured on main:
{code:java}
S3Client s3 = S3Client.builder()
.httpClientBuilder(Apache5HttpClient.builder().maxConnections(100))
// region, credentials, endpoint as usual
.build();
{code}
** On camel-4.18.x, use {{ApacheHttpClient.builder().maxConnections(...)}}
instead. This was checked in the source only.
** Also keep the routing of one poll under 60 s (a lower
{{maxMessagesPerPoll}}), or use {{includeBody=true}}: each body is then read
into a byte[] and its connection released.
* The description of the deprecated {{maxConnections}} points to "the
httpClientBuilder option on the S3 client configuration". camel-aws2-s3 has no
such option: it means the builder of the SDK S3Client, as above.
h3. Impact / upgrade
* The default {{includeBody=true}} does not exhaust the pool: each body is read
in {{createExchanges}} and its connection released. Only {{includeBody=false}}
with {{maxMessagesPerPoll}} above 50 does, or a lower value when other S3 calls
share the client.
* 4.18.5, 4.22.2 and 4.23.0 have the same consumer code, and an SDK with the 60
s default. The code was checked in the source, and the 60 s default in the
shipped sdk-core 2.41.28 and 2.50.2 binaries. It was reproduced on main only.
* A larger pool in the client Camel builds is a changed default and needs an
entry in the 4.24 upgrade guide.
h3. Related
* CAMEL-11697: the same failure in the SDK v1 S3 component, fixed in 2.20.0
with a larger pool.
* CAMEL-14929 added the stream body with includeBody=false.
* CAMEL-24157 (commit 4f887ea6d264) documented {{maxConnections}} as deprecated
with no effect.
* CAMEL-26662: the camel-aws2-s3 consumer improvement that gets each object per
exchange. It fixes this issue.
_Claude Code on behalf of Federico Mariani_
was:
h3. Problem
With {{includeBody=false}}, the consumer puts the open {{ResponseInputStream}}
of each object in the body. {{createExchanges}} gets every listed object of the
poll before {{processBatch}} routes the first one
({{components/camel-aws/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Consumer.java}}
lines 201, 298 and 207 on main, commit 7978db3d42dd). So all the streams of a
poll are open at once. Each holds a pooled HTTP connection until its exchange
reads it to the end or closes it.
* The SDK HTTP client pool has 50 connections by default, with a 10 s acquire
timeout ({{SdkHttpConfigurationOption}}, SDK 2.57.7).
* With {{maxMessagesPerPoll}} above 50, the 51st GetObject waits for a
connection: 10 s per attempt, 4 attempts.
* Since SDK 2.32.32, a {{ResponseInputStream}} that is not read within 60 s is
aborted. This is the default of {{ResponseTransformer.toInputStream()}}, which
the consumer uses.
* By the time the waiting GetObject calls give up, the first streams of the
poll are older than 60 s, and their exchanges fail with {{IOException:
Attempted read on closed stream}}.
* The failed objects are neither deleted nor moved, so the next poll lists them
again and the same thing happens on every poll.
* The {{maxConnections}} option (default 60, that is 50 + maxMessagesPerPoll)
has no effect: it is only a getter and a setter, deprecated on main
({{AWS2S3Endpoint.java}} lines 157-162).
* CAMEL-11697 fixed this exact problem in the SDK v1 S3 component by sizing the
pool at 50 + maxMessagesPerPoll. That fix did not carry over to camel-aws2-s3.
h3. How to reproduce
The setup is an in-process fake S3: a JDK {{HttpServer}} serving path-style
ListObjectsV2 and GetObject.
* Consumer:
{{aws2-s3://bkt?overrideEndpoint=true&forcePathStyle=true&uriEndpointOverride=...&includeBody=false&maxMessagesPerPoll=52}}.
* The route converts the body to String.
* The bucket holds 52 objects of 1 byte.
Measured on main with the client Camel builds:
* The poll took 60.2 s.
* The GetObject of the 51st object failed with {{SdkClientException: Unable to
execute HTTP request: Timeout deadline: 10000 MILLISECONDS ... (SDK Attempt
Count: 4)}}. It was skipped and given to the exception handler.
* 49 to 50 of the 51 routed exchanges failed with {{StreamCacheException ...
IOException: Attempted read on closed stream}} (several runs).
Control: the same setup with a user-supplied {{S3Client}} whose pool has 100
connections routed all 52 objects in 0.4 to 0.7 s, with no failure.
Separately, and with any pool size: with 2 objects and a route that takes 61 s
on the first, the second exchange fails the same way.
h3. Proposed change
* Best fix: get each object in {{processBatch}} just before its exchange is
routed, so that only the streams of exchanges in flight are open. This is the
improvement filed alongside. It also removes the 60 s abort of the later
streams of a poll.
* Otherwise:
** size the pool of the client Camel builds as CAMEL-11697 did (50 +
maxMessagesPerPoll);
** make {{maxConnections}} work again, or remove it;
** document that, with includeBody=false, the routing of one poll must finish
within 60 s.
* Workaround today:
** Supply your own client with a larger pool, through
{{amazonS3Client=#myClient}} or autowiring. Measured on main:
{code:java}
S3Client s3 = S3Client.builder()
.httpClientBuilder(Apache5HttpClient.builder().maxConnections(100))
// region, credentials, endpoint as usual
.build();
{code}
** On camel-4.18.x, use {{ApacheHttpClient.builder().maxConnections(...)}}
instead. This was checked in the source only.
** Also keep the routing of one poll under 60 s (a lower
{{maxMessagesPerPoll}}), or use {{includeBody=true}}: each body is then read
into a byte[] and its connection released.
* The description of the deprecated {{maxConnections}} points to "the
httpClientBuilder option on the S3 client configuration". camel-aws2-s3 has no
such option: it means the builder of the SDK S3Client, as above.
h3. Impact / upgrade
* The default {{includeBody=true}} does not exhaust the pool: each body is read
in {{createExchanges}} and its connection released. Only {{includeBody=false}}
with {{maxMessagesPerPoll}} above 50 does, or a lower value when other S3 calls
share the client.
* 4.18.5, 4.22.2 and 4.23.0 have the same consumer code, and an SDK with the 60
s default. The code was checked in the source, and the 60 s default in the
shipped sdk-core 2.41.28 and 2.50.2 binaries. It was reproduced on main only.
* A larger pool in the client Camel builds is a changed default and needs an
entry in the 4.24 upgrade guide.
h3. Related
* CAMEL-11697: the same failure in the SDK v1 S3 component, fixed in 2.20.0
with a larger pool.
* CAMEL-14929 added the stream body with includeBody=false.
* CAMEL-24157 (commit 4f887ea6d264) documented {{maxConnections}} as deprecated
with no effect.
* Filed alongside: the camel-aws2-s3 consumer improvement that gets each object
per exchange. It fixes this issue.
_Claude Code on behalf of Federico Mariani_
> camel-aws2-s3 - consumer with includeBody=false and maxMessagesPerPoll > 50
> exhausts the HTTP pool, most exchanges fail
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26661
> URL: https://issues.apache.org/jira/browse/CAMEL-26661
> Project: Camel
> Issue Type: Bug
> Components: camel-aws2-s3
> Affects Versions: 4.18.5, 4.22.2, 4.23.0
> Reporter: Federico Mariani
> Priority: Major
>
> h3. Problem
> With {{includeBody=false}}, the consumer puts the open
> {{ResponseInputStream}} of each object in the body. {{createExchanges}} gets
> every listed object of the poll before {{processBatch}} routes the first one
> ({{components/camel-aws/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Consumer.java}}
> lines 201, 298 and 207 on main, commit 7978db3d42dd). So all the streams of
> a poll are open at once. Each holds a pooled HTTP connection until its
> exchange reads it to the end or closes it.
> * The SDK HTTP client pool has 50 connections by default, with a 10 s acquire
> timeout ({{SdkHttpConfigurationOption}}, SDK 2.57.7).
> * With {{maxMessagesPerPoll}} above 50, the 51st GetObject waits for a
> connection: 10 s per attempt, 4 attempts.
> * Since SDK 2.32.32, a {{ResponseInputStream}} that is not read within 60 s
> is aborted. This is the default of {{ResponseTransformer.toInputStream()}},
> which the consumer uses.
> * By the time the waiting GetObject calls give up, the first streams of the
> poll are older than 60 s, and their exchanges fail with {{IOException:
> Attempted read on closed stream}}.
> * The failed objects are neither deleted nor moved, so the next poll lists
> them again and the same thing happens on every poll.
> * The {{maxConnections}} option (default 60, that is 50 + maxMessagesPerPoll)
> has no effect: it is only a getter and a setter, deprecated on main
> ({{AWS2S3Endpoint.java}} lines 157-162).
> * CAMEL-11697 fixed this exact problem in the SDK v1 S3 component by sizing
> the pool at 50 + maxMessagesPerPoll. That fix did not carry over to
> camel-aws2-s3.
> h3. How to reproduce
> The setup is an in-process fake S3: a JDK {{HttpServer}} serving path-style
> ListObjectsV2 and GetObject.
> * Consumer:
> {{aws2-s3://bkt?overrideEndpoint=true&forcePathStyle=true&uriEndpointOverride=...&includeBody=false&maxMessagesPerPoll=52}}.
> * The route converts the body to String.
> * The bucket holds 52 objects of 1 byte.
> Measured on main with the client Camel builds:
> * The poll took 60.2 s.
> * The GetObject of the 51st object failed with {{SdkClientException: Unable
> to execute HTTP request: Timeout deadline: 10000 MILLISECONDS ... (SDK
> Attempt Count: 4)}}. It was skipped and given to the exception handler.
> * 49 to 50 of the 51 routed exchanges failed with {{StreamCacheException ...
> IOException: Attempted read on closed stream}} (several runs).
> Control: the same setup with a user-supplied {{S3Client}} whose pool has 100
> connections routed all 52 objects in 0.4 to 0.7 s, with no failure.
> Separately, and with any pool size: with 2 objects and a route that takes 61
> s on the first, the second exchange fails the same way.
> h3. Proposed change
> * Best fix: get each object in {{processBatch}} just before its exchange is
> routed, so that only the streams of exchanges in flight are open. This is
> CAMEL-26662. It also removes the 60 s abort of the later streams of a poll.
> * Otherwise:
> ** size the pool of the client Camel builds as CAMEL-11697 did (50 +
> maxMessagesPerPoll);
> ** make {{maxConnections}} work again, or remove it;
> ** document that, with includeBody=false, the routing of one poll must finish
> within 60 s.
> * Workaround today:
> ** Supply your own client with a larger pool, through
> {{amazonS3Client=#myClient}} or autowiring. Measured on main:
> {code:java}
> S3Client s3 = S3Client.builder()
> .httpClientBuilder(Apache5HttpClient.builder().maxConnections(100))
> // region, credentials, endpoint as usual
> .build();
> {code}
> ** On camel-4.18.x, use {{ApacheHttpClient.builder().maxConnections(...)}}
> instead. This was checked in the source only.
> ** Also keep the routing of one poll under 60 s (a lower
> {{maxMessagesPerPoll}}), or use {{includeBody=true}}: each body is then read
> into a byte[] and its connection released.
> * The description of the deprecated {{maxConnections}} points to "the
> httpClientBuilder option on the S3 client configuration". camel-aws2-s3 has
> no such option: it means the builder of the SDK S3Client, as above.
> h3. Impact / upgrade
> * The default {{includeBody=true}} does not exhaust the pool: each body is
> read in {{createExchanges}} and its connection released. Only
> {{includeBody=false}} with {{maxMessagesPerPoll}} above 50 does, or a lower
> value when other S3 calls share the client.
> * 4.18.5, 4.22.2 and 4.23.0 have the same consumer code, and an SDK with the
> 60 s default. The code was checked in the source, and the 60 s default in the
> shipped sdk-core 2.41.28 and 2.50.2 binaries. It was reproduced on main only.
> * A larger pool in the client Camel builds is a changed default and needs an
> entry in the 4.24 upgrade guide.
> h3. Related
> * CAMEL-11697: the same failure in the SDK v1 S3 component, fixed in 2.20.0
> with a larger pool.
> * CAMEL-14929 added the stream body with includeBody=false.
> * CAMEL-24157 (commit 4f887ea6d264) documented {{maxConnections}} as
> deprecated with no effect.
> * CAMEL-26662: the camel-aws2-s3 consumer improvement that gets each object
> per exchange. It fixes this issue.
> _Claude Code on behalf of Federico Mariani_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
