Claus Ibsen created CAMEL-27180:
-----------------------------------
Summary: camel-aws2-kinesis - shardClosed=fail fails every poll
after a reshard, forever
Key: CAMEL-27180
URL: https://issues.apache.org/jira/browse/CAMEL-27180
Project: Camel
Issue Type: Bug
Components: camel-aws2-kinesis
Reporter: Claus Ibsen
h3. Problem
The polling {{aws2-kinesis}} consumer without {{shardId}} reads all the shards
of the stream. After a reshard ({{SplitShard}} or {{MergeShards}}) the parent
shard is closed, but it stays in the {{ListShards}} result for the retention
period of the stream (24 hours by default, up to a year). Since CAMEL-25569 the
consumer goes on with the child shards.
With {{shardClosed=fail}}, once the parent shard has been read to its end,
*every* later poll fails: {{handleClosedShard}} throws
{{IllegalStateException(ReachedClosedStatusException)}} inside the
{{parallelStream().forEach(...)}} of {{poll()}}, and the exception leaves
{{poll()}}. But the child shards are still read, as the other tasks of the
parallel stream keep running. So the option neither stops the consumer nor
reports the closed shard once; it logs a WARN with a full stack trace on every
poll until the closed shard leaves the shard list. Also:
* {{ForkJoinTask.invoke()}} returns on the first exception, so the next poll
can start while the shard tasks of the failed poll are still reading and
routing.
* the number of routed records of the poll is lost, and the
{{forceConsumerAsReady()}} at the end of {{poll()}} is skipped.
The option description is also out of date: it says that with {{ignore}} and
{{silent}} "the consumer will not process new messages until restarted", which
is only true with {{shardId}} set (without it, the child shards are read), and
it names the exception {{ReachedClosedStateException}} instead of
{{ReachedClosedStatusException}}.
h3. How to reproduce
Stream {{w6split}} with one shard, a few records, then {{aws kinesis
split-shard}} into shards 1 and 2 (tested with Camel 4.24.0-SNAPSHOT against a
local Kinesis emulator):
{noformat}
aws2-kinesis://w6split?iteratorType=TRIM_HORIZON&shardClosed=fail&delay=500&...
{noformat}
The first poll routes the records of all three shards. From then on, every 500
ms:
{noformat}
WARN Kinesis2Consumer : Failed polling endpoint:
aws2-kinesis://w6split?...&shardClosed=fail...
java.lang.IllegalStateException: java.lang.IllegalStateException:
org.apache.camel.component.aws2.kinesis.ReachedClosedStatusException
...
at
org.apache.camel.component.aws2.kinesis.Kinesis2Consumer.poll(Kinesis2Consumer.java:146)
Caused by: java.lang.IllegalStateException:
org.apache.camel.component.aws2.kinesis.ReachedClosedStatusException
at
org.apache.camel.component.aws2.kinesis.Kinesis2Consumer.handleClosedShard(Kinesis2Consumer.java:365)
at
org.apache.camel.component.aws2.kinesis.Kinesis2Consumer.getShardIterator(Kinesis2Consumer.java:279)
{noformat}
29 such failures in 15 seconds; records written to the child shards in that
time were still routed.
h3. Where
*
{{components/camel-aws/camel-aws2-kinesis/src/main/java/org/apache/camel/component/aws2/kinesis/Kinesis2Consumer.java}}:
{{poll()}} (the {{parallelStream().forEach}} over {{getCurrentShardList()}}),
{{getShardIterator(...)}} (calls {{handleClosedShard}} on every poll for a
closed shard), {{handleClosedShard(...)}} ({{case fail}}).
*
{{components/camel-aws/camel-aws2-kinesis/src/main/java/org/apache/camel/component/aws2/kinesis/Kinesis2Configuration.java}}:
the description of {{shardClosed}}.
* The only test, {{KinesisConsumerClosedShardWithFailTest}}, has a single shard.
h3. Suggested change
Decide what {{fail}} means when all the shards are read. For example:
* report the {{ReachedClosedStatusException}} once per closed shard (through
the consumer's {{ExceptionHandler}}, so {{bridgeErrorHandler}} works), and then
skip that shard like {{silent}}, while the other shards go on; or
* keep the current meaning (stop with an error) only with {{shardId}} set, and
treat a closed parent shard with known children as normal resharding.
In either case, do not throw out of the parallel stream while other shard tasks
run (see also CAMEL-26670, which moves the shard reads to a Camel-managed
executor), fix the option description, and add a multi-shard test.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)