[
https://issues.apache.org/jira/browse/CAMEL-24197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-24197:
--------------------------------
Fix Version/s: 4.22.0
> camel-aws2-kinesis - Fixed-shardId DescribeStream on every poll and batch
> partition key
> ---------------------------------------------------------------------------------------
>
> Key: CAMEL-24197
> URL: https://issues.apache.org/jira/browse/CAMEL-24197
> Project: Camel
> Issue Type: Bug
> Components: camel-aws
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Minor
> Labels: review-2026-07
> Fix For: 4.22.0
>
>
> Split from CAMEL-24157 (medium-severity findings, group 3).
> Two remaining Kinesis findings:
> *1. Fixed-shardId consumer calls DescribeStream on every poll*
> {{Kinesis2Consumer:108-141}} — When {{shardId}} is configured (non-empty),
> every call to {{poll()}} issues a {{DescribeStream}} API call to locate the
> Shard object. Two problems:
> - Rate-limit risk: DescribeStream has a 10 TPS per-account limit. With
> default 1-second poll delay, a single consumer approaches the limit; multiple
> consumers exceed it.
> - >100 shard failure: DescribeStream returns at most 100 shards per response.
> The code does not paginate (no {{exclusiveStartShardId}} loop), so for
> streams with >100 shards the configured shard may not be found, throwing
> {{IllegalStateException}}.
> The multi-shard path correctly uses a cached {{ShardMonitor}} with the
> {{ListShards}} API (paginated, higher rate limit). The fixed-shardId path
> bypasses this caching entirely.
> *2. Batch producer applies single partition key to all records*
> {{Kinesis2Producer:77-122}} — A single {{CamelAwsKinesisPartitionKey}} header
> is applied to every {{PutRecordsRequestEntry}} in the batch. All records are
> routed to the same shard. This is a design limitation — no per-record
> partition key mechanism exists in the current API.
> *Fix approach:*
> - #1: Cache the shard iterator for the configured shardId (similar to
> ShardMonitor), avoiding DescribeStream on every poll. Add pagination for
> >100-shard streams.
> - #2: Design decision needed — consider accepting
> {{List<PutRecordsRequestEntry>}} directly, or a wrapper type carrying
> per-record keys, or deriving partition keys from each record body.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
